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INTRODUCTION 

This Notebook is a collection of Western Regional Center newsletter articles and workshoD 
matenals on evaluating alcohol and other drug (AOD) prevention programs. It is a revised and 
condensed version of a notebook originally assembled in 1990. The purpose of the Notebook is 
tohelp individuals who are beginners in program evaluation to start a notebook of ideas and 
strategies on evaluaung AOD programs. With a 3-ring binder, other handouts and materials on 
evaluauon can be easily inserted for continual updating. 

The current Notebook contains the following sections: 

(1) Evaluation Issues 

(2) Evaluation E*ractice 

(3) Needs Assessment 

(4) Program Planning and Process Evaluation 

(5) Outcome Evaluation 

(6) Evaluation Resources 

(7) Evaluation Instruments 

^ • °" ^^"^f^ evaluation topics and strategies. Sections 3, 4, and 5 reflect 

Sfrif^^ ^ " evaluation cycle which begins with identification of a need and ends 

with the assessment of the results obtained by the program put in place to meet that need 

f^^Z. f"'™?.^^^ ^ """i^r 0/ resources in evaluation and Section 7 provides two evaluation 
instruments, which may be reproduced for use. oiuauuu 

The goal of the Notebook is to make it a readily available resource tool for AOD coordinators 
and school-community team members involved in the ongoing process of planning 

^"^ ^? programs. Readers should refer to existing manuals or 

textbooks on evaluation (see Sec.^on 6) and seek technical assistance from professional 
evaluators for situations that call tor rigorous methodologies. 
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Section 1 

Evaluation Issues 



SECTION 1. EVALUATION ISSUES 



The readings in this section address overall issues in evaluation. This section begins with an 
article describing a comprehensive model for drug-free schools and communities, in which 
evaluation is seen as one of many connected components. This is the approach advocated by the 
Western Regional Center. Several additional readings from the Western Center News and 
exhibits adapted from workshop presentations are also included. 

The Western Regional Center for Drug-Free Schools and Communities takes the position that 
evaluation is an integral component of a comprehensive approach in program development. In 
other words, evaluation is not seen as an afterthought at the end solely to fulfill funding 
requirements. 

As depicted in the diagram on page 1-5, evaluation is part of an ongoing cycle of planning, 
implementation, and refinement. Results from evaluation serve two useful purposes. The 
information collected can be used to identify program strengths and weaJcnesses for further 
improvement. It also can be used to disseminate program features and accomplishments to 
interested audiences such as school boards, district administration, school staff, students, parents, 
and business or community representatives. 
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Drug-Free Schools and Communities - Making a Real Difference 

(From Western Regional Center, Northwest Regional Eoucational Laboratory, Planning for Drug-Free Schools and 

Communities: Portland, OR. September 1991) 



With proper planning, tools, and information, schools and community representatives can 
make a difference in providing students and teachers what they have a right to expect - a 
drug-free and safe environment in which to learn and to teach. Planning will also extend to 
community activities in order to support, supplement and sustain the activities in the schools. 

In enacting the Drug-Free Schools and Communities Act, the Congress recognized that drug 
and alcohol prevention, education, and intervention programs are essential components of a 
comprehensive strategy to reduce the demand for and use of alcohol and other drugs (P.L. 
101-226). This strategy was incorporated into a comprehensive program planning and 
development model in 1987 (Fox, Forbing, Anderson, 1988) that has served as the model for 
the training and technical assistance of the Western Regional Center. This strategy is 
beginning to work in the Western region as evidenced by the development of greater 
awareness of the harmfulness of drugs and alcohol for children and adolescents among 
school and community leaders and the resultant reductions in use of most drugs, 

Tlie mission of the Western Regional Center is to provide training, technical assistance, and 
dissemination of information services to educational and community personnel based on 
current theory and research applications to prevent the illegal use of alcohol or other drugs 
among children and adolescents, and to reduce the harmful effects of use and abuse among 
those who have. 

The Western Regional Center's approach for carrying out this mission is based on a 
prevention philosophy; a model or framework for community planning, decision making and 
action; and a process for providing training and technical assistance toward the goal of drug- 
free schools and communities. 

This philosophy is based on current theory and research in the causes and effects of alcohol 
and drug use and abuse, and necessary ingredients for effecting change in local schools and 
communities. These are detailed below. 

The Effects of Alcohol and Other Drug Use. Central to planning and implementing 
programs is the conviction that any use of alcohol or drugs by children and adolescents is 
illegal and harmful (P.L. 101-226), and that use of alcohol and other drugs nmy cause 
physical and emotional dependence (USDOE, 1989). Other principles guiding the Western 
Regional Center's training and technical assistance are: 

■ The earlier the alcohol ana other drug use, the more likely the dependence (Rachel et, 
al., 1982. Fleming et. al., 1982; Hawkins, et,a]., 1986)) 

■ Early use of gateway drugs (tobacco, alcohol, and marijuana) may lead to use of other 
drugs (Kandel, 1978; Austin, 1988; Deck and Nickel, 1990) 

m Children of substance ui>ing and abusing parents are themselves at higher risk of use 
and abuse (Bushing and Bromley, 1975; Hawkins, et.al., 1986; Austin and 
Prendergasi, 1991) 

■ Without intervention ulcohol, and other drug dependency is progressive (MacDonald, 
1984: Schaefer, 1987) 
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■ Alcohol and other drug dependency is a treatable problem (Johnson, 1980; Wallack, 



Multiple Causes of Use and Abuse. The development of prevention strategies must be 
based on solid theory and research which, at this time, are still emerging. Successful 
approaches must recognize the multidimensional etiology of alcohol and drug use and abuse, 
and counter with prevention strategies that include the interacting individual, community, 
and substance factors which lead to use (OSAP, 1989). Individual factors include: 

■ Genetics (Cloninger, et al., 1986) or the influence of living in a home of substance 
abusing parents (Kumpfer, 1987; Kumpfer, 1989) 

■ Personality (lessor and lessor, 1977; Shedler and Block, 1990) including risk and 
protective factors (Hawkins et.al., 1986; Benard, 1987; Benard, Fafoglia and Perone, 
1987; Garmezy and Rutter, 1983; Rutter, 1984) 

■ Attitudes and belief systems (Johnston, O'Malley and Bachman, 1989; Norem- 
Hebeisen and Hedin, 1987) 

Community factors recognize that behavior is highly influenced by the social context of 
values, customs, norms, cultures, economic climate, deterrence activities, ana attitudes of the 
community (Perry, 19S6), and by family, peer, school, and neighborhood influences 
(Bronftenbrenner, 1979; Hawkins, et al., 1985; Hansen et al, 1988) 

The substance itself, including availability, accessibility, and toxicity of the substances, also 
interacts to determine use, abuse, and dependency (Coate and Grossman 1987- Wallack 
1985; Moskowitz, 1989) 

Multiple Systems Changes Needed. A comprehensive, multi-system, multi- strategies 
(individual, family, school, community) approach to prevention and early intervention is 
necessary. The problems of alcohol and drug use and abuse are not simply those of 
individual weakness, as noted above. They are highly systematic. Efforts at change must be 
directed toward school, community, and societal systems (Schaps, Moskowitz, Malvin and 
Schaeffer, 1986; Bell and Battjes, 1985; Earle, 1984; OSAP, 1989; Hawkins et al., 1986- 
Gibbs and Bennett, 1990; Muggerty, et.al., 1989). 

Empowering School-Community Teams. Studies of social change indicate the impetus to 
change and solutions to the alcohol and other drug problems lie within the community 
Teams of individuals can be taught to analyze, plan for, implement, and effect changes 
(Allen and Allen, 1987; Benard, 1989; Earle, 1984; Pentz, 1986). While outsiders may be 
change agents, they serve best to guide and offer alternatives and opportunities to network 
The work itself must be done by school-community teams. 

Comprehensive Approach to Training School-Community Teams 

An appropriate model or procedure based on the latest theory and research is needed for the 
Western Regional Center to provide training and technical assistance to schools and 
conirnunities. The Western Regional Center implemented such a model in 1987 (Fox 
Forbmg, and Anderson, 1988), and has used it for the past three years in training more than 
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Se ction 1 Evaluation R asource Notebook 

2,000 school-community teams. It has proven valuable in setting forth both the process and 
the content for implementing drug-free schools and communities programs.* 

The model has been progressively revised and refined, based on the latest research and the 
wealth of field experiences accumulated by the Western Regional Center in working with the 
complex processes operating at state and local levels. Figure 1 on the following page depicts. . 
our adaptation of the Fox and Forbing model. The key processes of identifying an iniual 
planning group; assessing needs; planning programs in prevention, early mtervenuon, 
referral, and support services; evaluating program progress; and disseminaung results have 
not changed What has changed is the context in which planning and implementation occur. 
They occur within the context of a multi-layered social system that includes schools, 
communities, and families. 

Representatives of each of these systems must be a part of the initial planning, as well as the 
implementation of ongoing activities throughout the process. Qear and comprehensive 
school and social policies are critical foundations to all planning and implementauon 
activities. Prevention and early intervention activities must focus on fanuly and community, 
as well as school environments. Active dissemination must occur at all points of the process- 
-from initial planning, needs assessment, and specific program planning through the 
evaluation of program effects. The model represents an ongoing process, in that it 
emphasizes an information flow that continually feeds into planning and new or revised 
program implementation. 

Each component of the model is described in greater detail below, including research-based 
rationale and expected outcomes. 

Initial Planning and Training. The readiness to begin a process which will impact alcohol 
and other drug use and abuse in the community begins with an interested group of 
individuals who are willing to take on the leadership needed to initiate the comprehensive 
effort. If the impetus comes from the Drug-Free Schools and Communities Act, this will 
likely be a group from the school or district. If the impetus is in response to the emphasis on 
community prevention that the Office of Substance Abuse Prevention (OSAP) has taken, the 
group may be primarily elected officials or a grass-roots community group. Regardless of 
the group's genesis, the role of the Western Regional Center is to provide training and 
technical assistance based on research evidence and delivered within a framework for acuon 
that can be used by the group to assist in meeting their objectives. The most effective teams 
include diverse representation of families, religious institutions, schools, government, 
business and industry, law enforcement and justice systems, media, health and social service 
systems (Rothman, 1979, Brown, 1984, Pentz, 1986). 



•This discussion is based ujwn the Fox and Forbing model (1991). 
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Section 1 Evaluation Resource Notebook 

The purposes of training and technical assistance provided by the Western Regional Center in 
assisting school-community teams plan and implement a comprehensive program are to raise 
awareness, spread consistent messages, reduce denial, develop team spirit, and develop an 
understanding of the planning and implementation process and content necessary to achieve a 
drug-free school and community. 

Teams need to be empowered through identifying their purpose and working through a 
procedure or model (Mink, Mink and Owen, 1987). The Western Regional Center provides 
initial team training in the process for system change- "Planning for Drug-Free Schools and 
Communities." The focus of this training is to enlarge the initial planning team to include all 
relevant community stakeholders, and for the team to gain an understanding of the planning 
process and the multiple strategies necessary for a comprehensive approach. According to the 
\].S. Department of Education, "to combat student drug use most effectively, the entire 
community must be involved: parents, schools, students, law enforcement, religious groups, 
social service agencies, media, and business" GJSDOE, 1987, 1989). 

It is widely recognized that issues related to program implementation can be a key factor in the 
success of a program (Tobler, 1986; Goodstadt, 1986; Moskowitz and Smith, 1987). As 
program strategies are established in the areas of prevention and early intervention, additional 
training is necessary to ^.nsure program success. In the area of school-based prevention 
programs, for example, ongoing in-service training has been shown to be a primary factor related 
to teachers* ability to deliver a credible message and implement the cuniculum appropriately 
(Klitzner and Bell, 1987; Lohrman and Fors, 1986; Benard et al., 1987). The appropriate 
implementation of curriculum is not simply a matter of fidelity to the author's directions. The 
local context must weigh heavily, including cultural and economic factors (kobels, 1989). The 
Western Regional Center approach includes providing the additional training for teams to 
sequentially implement their comprehensive plans. 

Outcomes of initial team formation and training include awareness of the complex and 
interacting issues in the etiology of alcohol and other drug use and abuse problems; the 
commitment to develop comprehensive plans and programs within the school-community 
geographic area; an understanding of an array of program strategies appropriate to eliminating 
alcohol and otiier drug use and abuse among children and adolescents; and an understanding of 
the role of the team in implementing programs in the short term and in changing community 
systems in the long term. 

Needs Assessment. The pu^pos^r of needs assessment is to assess and understand the nature and 
scope of alcohol and other drug problems in the local community in order to, first, raise 
awareness that the "problem is here" and, second, to match local needs to the development and 
implementation of program strategies. Western Regional Center training provides the school- 
community teams with a variety of data gathering and interpretation activities used to guide the 
development of policies and programs. It is a key step in the process-one which, if not done 
well, can put the local school-community team through misdirected or wasteful programmatic 
efforts. As noted by OS AP: "All prevention programs require knowledge of the target 
community and the context in which alcohol and other drug use occurs. In addition, school 
Personnel should know the extent of alcohol and other drug use in their schools..." (OS AP, 

The outcomes of a needs assessment process are data and reports that can be used to combat 
denial and educate the community about the extent of the alcohol and other drug use and abuse 
problem and potential relationships to other risk behaviors (e.g., teen suicide, dropouts). Needs 
assessment da^a can also include an outline of available resources. Finally, needs assessment 
data provide a baseline from which subsequent program progress can be measured. 
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Policy Development The development of sound policy-both fnnnal (e.g., school policy laws 
S2lSlfn?nl'"'* mformaKnorms, values, counseling, and warning -is essentials the ' 
development of a comprehensive approach. The puipose of policy is to have a consistent 
?nS:;r.l!'?- ' "^I^'T ^^^^^^ ^"^ ySuth that is reinforced by aU socS 
f„H ?rSi"fn^^fn r"'^- School policy lays the foundation for a comprehensive discipline 

TfJIZ^T f?l'°^''^ ^^^^ '"PP°^ well-being of the entire school community As 

^Tt^l^^- ^^Pf^^"^ of Education. "School policies should clearly establish Sat d^g 

mi^S Zi\'^'?u'^^ "^^^^ ^^^'^'^^ ^hool functions will not be tolemed^^ 
(.U5UUfc, 1987). These policies, while stressing consequences of use, also mandate the 
aeveiopment of a comprehensive prevention and intervention strategy (P.L. 101-226). 

Policy development is not a one-time activity. Formal and informal policy adjustments must be 
made by school-community teams as they assess their community's response to the alcohol and 
other drug problem. For example, lax or inconsistent enforcement of n§nor-in-poSon in^he 
school or commumty statutes will give the mixed message that using illegal sSbsSTs Jkav 
Sldt^ oSle."'^'' '""'^ ^ strengthened and key^segments of the coZSiiy 

The outcomes of poUcy development will include the installation of comprehensive school 
S£'n'.r/''''";y enforcement of local ordinances, and the development of a co^un ty 
ordinance supporting a no-use stance for children and youth. "wununuy 

Program Planning. Program planning is the process whereby the team establishes goals 
SSh ' i-^Ple-^entation of program strategies that are based on 

idenuf ed needs, and are selected as appropriate for their community. The planning includes 

C^v£Sn"^e;Slt^i' '"^'"^^^ populations that impact children and adollscents a^d 
invokes muluple strategies in prevention, intervention, referral, and support. 

The outcome of the program planning is a team action plan with objectives, activities time lines 
and funding sources identified. The plan guides the implementatioil prograi^ ' ' 

sS'?FoTJ,d Fo^h^^^^^ of prevention is to cairy and reinforce the message, "Don't 

iM^^^o^n^^^i^^^^^ - - °f ^he multipleUms 

Prevention strategies focusing on the community system include: 

■ fnl m&^L^,^^^"^"/!^^ segments of the community regarding the nature of alcohol 
and other drug use and abuse among children and adolescents 

■ v!.^"'* organizing with other organizations within the community to achieve 
muluple, yet non-redundant goals avi.itvc 

■ 2Zwf v^r^r""'!,^ values and norms directly through social and public policy (e.g., 
alcohol taxes), or indirectly through impacting attitudes about the hirmfulness of early 

■ fi^^^^*/ alternatives to alcohol and other drug use and abuse, and support 
services for families m need, including health and economic resources 

'^'.^^^Sies focusing on the family systems include parent education and self-heb 
groups whereby parents are encouraged to understand the: ^ 

■ Nature and extent of the alcohol and other drug use problems in their community 
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m Harmfulness of early alcohol and drug use 

■ Risk and protective factors associated with use ^ 

■ Family management skills 

■ Communication skills, including the importance of a clear no-use message 

■ Signs and indicators of early use • 

■ Resources available for help (Bell and Battjes, 1985; Brook et al., 1986; Kumpfer, 1987; 
Hawkins et al., 1986; Bennett et al., 1988) 

Family prevention strategies require sensitivity and modification to reach families who do not 

typically participate in such efforts, even when made available to them. Strategies focused on # 

the family can provided by school or community organizations. 

Prevention strategies focusing on the school include providing education for school personnel, 

parents, and students. Students need a comprehensive, developmentally based, age-appropriate 

prevention and education program in every grade that carries the consistent message that alcohol 

and other drug use is wrong and harmful. Tlie education and prevention programming should be • 

directed at: 

■ Enhancing personal, social, and refusal skills 

■ Early reinforcement of non-use messages 

■ Providing factual information 

■ Reinforcing ihe fact that alcohol and other drug use are risks to children and adolescents 

■ Restructuring the school environment to prevent early antisocial behavior (Goodstadt, 

1989; Weisheit, 1983; Malfetti, 1983; Klitzner, 1987; Pisano and Rooney, 1988) # 



While the classroom is an extremely important place for prevention activities, expanding beyond 

its limits can complement and strengthen the school's efforts to impact student behavior (Bickel, 

1989). Examples of non-classroom activities include after- school programs, teen conferences, 

peer leadership programs, awareness raising events such as Red Ribbon Week, and alcohol and 

other drug-free events. # 

In addition to activities focused on students, schools can provide training for parents and 
teachers in overall awareness and understanding of the needs and resources available to prevent 
use of alcohol and other drugs. Teachers need additional training in providing the array of 
techniques and skills required for the current generation of prevention programs (OSAP, 1989). 

• 

Tlie outcomes of commlinity-, family-, and school-based prevention programs include increased 
awareness among all of these audiences of the dangers of alcohol and drug use by children and 
adolescents; the implementation of specific strategies in each of these systems to reduce risk and 
increase protective factors; an increase in healthy alternatives; and changes in norms, values, and 
expectations in communities, families, and schools. 

• 

Early Intervention Strategies. The purpose of early intervention activities is to carry and 
reinforce the message, "Stop Now (Fox and Forbing)." Eariy intervention with ch.^dren at high 
risk of use in the elementary grades (Kellam, Ensrninger, and Simon, 1980; Block, Block, and 
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Keyes, 1988), and identification of those who may already be involved with alcohol and other 
drugs is an important strategy to stem the progression to problem use and dependency 
(Anderson, 1987; U.S. HHS., 1984; Milgram, 1989). 

To reach the largest number of affected children and adolescents, early intervention programs 
should be offer^ in the school system. Early intervention is a key component of a Student 
Assistance Program (SAP), modeled after the Employee Assistance Programs in business and 
industry. The increased development and implementation of SAPs in the school systems in the 
Western region represents one of the most significant "systems" change noted over the past three 
years. Rather than taking merely a disciplinary approach to student use and abuse of alcohol and 
other drugs, teachers and administrators are beginning to see their role as identifying and 
interrupting the continuation of harmful and illegal behavior. 

Intervention services are provided in the community and through the family as community and 
family norms are sharpened to focus on early identification and referral to appropriate help. 
Training of students, families, and members of youth-serving organizations (e.g., scouts, 
YMCAs, community athletic leagues) to identify high risk children and identify the signs and 
symptoms of use is an integral part of a comprehensive program (Klitzner, and Bell, 1987; 
Kantor,etaL, 1990). 

The outcomes of intervention include formal processes within schools (such as SAPs) whereby 
children are identified as high risk for use or as early users, and are provided appropriate school- 
based support and community-based services. Other outcomes include increased awareness 
among those who wcrk in the community with youth and among families of the signs of high 
risk and/or early use. Finally, all systems-school, family, and community- will have the skills 
to identify and intervene with children and youth. 

Referral Strategies. Referral strategies include the development of procedures for referring 
affected children and youth to community-based services for assessment and treatment of 
alcohol and other drug problems. The message of referral services is "We Clan Help." Schools 
work with community-based agencies to develop and disseminate the working arrangements and 
maintain up-to-date directories of resources and services available. Communities, families, and 
schools can work to ensure that these services are available and accessible to children and 
adolescents in the communities (Anderson, 1987). 

The key outcome of referral strategies is the appropriate matchmg of children and adolescents to 
needed services. There must also be a shared awareness among all systems of standard 
procedures for referral and the array of resources and services available. 

Support Strategies. Support strategies are those that provide the opportunity for groups of 
peers to reinforce their education about the nature of alcohol and other drug use, and encourage 
positive and healthy living through practice in communication, coping, decision making, and 
refusal skills. Support groups may be provided by school systems for a wide spectrum of 
children-those at high risk of use, those who have used, those who are concerned about 
another's use, and those who have used and have stopped (Anderson, 1987; NIAAA, 1984; 
Robinson, 1989). 

Community systems provide support strategies by encouraging the development of self-help 
groups for chUdren and adolescents, such as Alcoholics Anonymous, Alcoholics Victorious, 
Narcotics Anonymous, Alateen, Alatot, etc. Communities can also provide incentives or policies 
that direct treatment agencies and other social and health service providers to expand their 
aftercare and support services. 
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The intended outcome of effective implementation of support strategies is a system of support 
that is easily accessible to all interested children, youth, and families. Such a system is 

characterized by support groups in all schools, elementary through high school, and a range of ^ 
community support groups, including extended care through treatment organizations. 

Evaluation. An ongoing system of feedback and revision/refinement of program activities is an 

essential component of any comprehensive program (Armstrong et al., 1986). Formative or 

process evaluations document milestones of the planning process, program implementation and 

costs, numbers served, time spent, and informal reports of children and adolescent responses to • 

programs. 

Summative evaluations include measures of the expected outcomes of the program strategies- 
changes in attitudes, knowledge, and behaviors of students, families, school personnel, or 
members of the community (Hawkins and Nederhood, 1987). In addition to measures of use of 
alcohol and other drugs, measures of other related behaviors may be used to evaluate the • 
effectiveness of the school, community, and family programs. Community indicators include 
rates of youth suicide, homicide, drug-related arrests, and drug-related hospital emergencies. 
School indicators include dropouts, attendance, tardies, and disciplinary referral (Deck, 1988). 
These indicators are often included in the needs assessment phase and their continual monitoring 
provides useful evaluation information. 

• 

The outcomes of ongoing formative and summative evaluation activities help the planning teams 
to identify needed resources-both financial and technical-staffing or program adjustments, and 
modifications in school-community-family system relationships. The evaluation component 
serves as a self-correcting mechanism to support the ongoing planning and implementation of 
services (Armstrong et al., 1986; Hord and Loucks, 1980). 

• 

Dissemination. Active dissemination assures that the progress made by Uie schools and 

communities in achieving drug-free schools and communities can be shared with others. It is a 

process which should be carried out at every phase of planning and implementation, since it 

serves as an awareness and norm-changing process as the communities and schools become 

more informed of the efforts of the teams. As programs are implemented and revised or refined, 

information should be provided to the families, schools, and communities. Dissemination can # 

occur in a variety of ways— written documents, presentations before community and school 

groups, advertising or news coverage in the media, and special events such as art projects. 

Successful programs must to be documented so that other communities can review their 

processes, target groups, objectives, activities, and evaluation results for potential replication. 

Exemplary programs are showcased through Western Regional Center planning and networking 0 

activities. 



The outcomes of dissemination aie increased community knowledge and attitudes reflecting 
greater understanding of the issues related to youth alcohol and other drug use. These produce 
further awareness in prevention and intervention program strategies. The message of 

dissemination is that local conuTiunities have the problems associated with alcohol and other # 
drug use, but also have the commitment and resources for planning, implementation, and 
ultimately, attainment of drug-free schools and communities! 

The outcomes of Western Regional Center training and technical assistance for each component 
of the model described above are presented in Table 1 . 
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Table 1 



Model Component 
Initial Training 



Needs Assessment 



Program Planning 



Prevention 



Intervention 



Referral 



Support 



Evaluation 



Dissemination 



Policy 



Training and Technical Assistance Outcomes for Each Component of the Model 

Training and Tech nical A<;5;i;gtance Outcomes 

Awareness of complex issues in etiology and 
solutions 

Commitment to add community team members 
Commitment to continue with planning process 
Initial action plan 

Data and reports available for each community to 
be used in dissemination and planning 

Installation of comprehensive school policies 
Consistency of local law enforcement 
Community policies supportive of no use for 
children and adolescents 

Increased awareness of alcohol and other drug 
issues among families, schools and communities 

Specific prevention strategies implemented by 

families, schools, and communities 
Changes in norms about use and abuse of alcohol 
and other drugs by young people 

Formal identification and intervention process in 
schools 

Increased awareness among schools, families, and 

communities of signs and symptoms of abuse 
Formal intervention systems available in the 
community 

Appropriate matching of children and youth to 
community assessment and treatment resources 

Self-help groups available in the community 
Follow-up care available in the community 

Formative and summative evaluations completed by 
each school-community team 

Increase in community knowledge about alcohol and 

other drugs 
Increase in knowledge about prevention and 

intervention strategies 
Shared program strategies among school-community 
teams 

Increase in noteworthy programs 

Components of an effective school substance use policy 
Experience in writing policy 
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Relationship Between Development Process 
and Content of Comprehensive Program 

Training and technical assistance guided by the Western Regional Center model stresses the 
teams' understanding of both the development process and the content of a comprehensive 
program. 

Training workshops have been and will continue to be developed to assist teams in acquiring the 
skills or in identifying other appropriate resources in specific focus area, i.e. the content. Each 
training in a content segment (e.g., Curriculum or Support Group development) reinforces the 
need for that effort to be articulated with the overall process, i.e., initial planning and needs 
i\ssessment through dissemination, so that implementation does not occur in a vacuum. 

W5th this dual emphasis teams are better able to recognize their roles from either viewpoint: that 
ther content area must proceed through a recognized process to ensure success, and that any one 
task of the process (e.g., needs assessment) is a necessary agenda item to successfully implement 
content areas. Attention to both content and process by all interested parties (schools, families, 
and community) brings unity and moves consistently toward new norms and institutionalization. 
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Porcupines Making Love 

(There's a lot of promise, but it sure seems 

risky trying it.) 



Image of Evaluation from: Handbook for Evaluating Drug and Alcohol Prevention Programs (STEPP) by 
J, David Hawkins and Brilt Ncderhood (1987). Drawing by Marjoric Wolfe, Western Regional Center. 



Evaluation: A Shared 
Responsibility 



By ROY M. GABRIEL 

Northwest Regional Educational Laboratory 

What images does •'evaluation*' conjure up 
for you? Is it like filling out yoxir income tax 
form when you know you're not getting a 
refimd? Or perhaps it's reminiscent of the sex 
life of porcupines-it looks sort of interesting, 
could be pronusing, but it's very risky business! 

In the battle to rid our schools and 
conununities of illicit drugs and alcohol abuse, 
we all share the responsibility for looking 
objectively at what we're doing and trying to 
improve our efforts. Evaluating the 
effectiveness of our programs is an important 
and integral component of these efforts. 

Evaluation can take on a variety of purposes 
and procedures, and be directed to any of a 
number of interested audiences. Think for a 
minute about the different groups that nught be 
interested in information about your program. 
Parents, school staff, staff in neighboring schools 
and districts, state administrators, potential 
funding agencies at federal and state levels, and 
the news media are concerned with knowing the 
results of drug and alcohol progranns in your 
school. 

The Western Center offers a full-day 
workshop for school and district staff on 
Evaluating School Prevention <ind Intervention 
Programs. In it, a general process for evaluating 
your programs is described, and ample time for 
working with school teams to devise tlieir own 
evaluation strategies is provided. Sample 
surveys of student alcohol and drug use are 
made available to participants, and a prototype 
recordkeeping system for student assistance 
programs is presented. 

A key concept in getting ready for a 
meaningful and useful evaluation is embodied 
in a research term known as evaluability . That 
is, what are the necessary ingredients your 
program must have to be ready for an 
evaluation? 



Is the program sufficiently formulated and 
implemented to look at its impact on kids' 
behaviors? Or is it only in its begiiining stages, 
in which case a process of internal monitoring to 
help fine*tune the activities would be more 
useful? 

Think carefully about where you are in your 
program's implementation and plan yoiir 
evaluation accordingly. The checklist 
reproduced here should help you. If you find 
yourself responding TJo" to any of the items, it 
is recommended that you txim your attention to 
those areas. The goal is to be able to respond to 
all ten items with a Tes," 







Are 


You Ready To Evaluate? 


Yes No 






( ) 


( ) 


1. 


Are all proeram components 
dearly described? 


( ) 


( ) 


2. 


Do all components have defined 
goals and objectives? 


( ) 


( ) 


3. 


Are these goals and objectives 
grounded in an objective needs 
assessment? 


( ) 


( ) 


4. 


Do program staff and 
policymakers agree that these 
objectives are realistic? 


( ) 


( ) 


5. 


Are the program's activities 
logically consistent with its 
goals? 


( ) 


( ) 


6. 


Have all program components 
been implemented? 


( ) 


( ) 


7. 


Can valid, n^easurable 
indicators be obtained or 
devised for each objective? 


( ) 


( ) 


8. 


Is it financially feasible to collect 
these data? 


( ) 


( ) 


9. 


Do program staff know what 
kind of information an 
evaluation can produce? 


( ) 


( ) 


10. 


Is there a commitment on the 
part of program staff <ind school 
administrators to disseminate 
and use the results of the 
evaluation to improve the 
program? 
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By JUDITH A. JOHNSON 

Northwest Re^ond Educational Laboratory 

Don't fo]*get the conununity when putting 
together the needs assessment/evaluation plan 
for yoiir school. Without looking at the 
community's perceptions you won't have a 
comprehensive view of your program 
effectiveness, let alone have any imderstanding 
of the relevance of your school program within 
the T>ig picture" of the entire conununity, . 

YouH need to tap into all of the information 
surrounding your school-resources, law 
enforcement support, parental attitudes, 
cultures, values, dvic responsiveness, and 
religious views in order to plan for a drug-free 
environment for the students and for the 
commimity. 

You'll need to reach out and discover what's 
influencing the student behavior /use/abuse- 
current fads, popular movies, popular music, 
and new drugs coming into the community. Is 
alcohol being sold by the same store ov/ners, or 
mostly being obtained through the family's 
supply? 



Where are students using and when? These 
are two vital questions when you begin to 
examine alternatives. Who are the kids' role 
models, not only in the school but within the 
community? 

Soimd like lots of ii\formation to gather? 
Most of it will come from the students 
tiien\selves if you provide a safe environment 
for an open discussion. A great deal will come 
from observations and open fonmis with 
parents. Qvic organizations and law 
enforcement will cooperate too, if they know 
why. 

You're looking for impact, you're looking for 
a comprehensive program, and you're looking 
for people to support the program. Utilizing 
needs assessment and evaluation will provide 
you with the mechanism you need. 
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Task Force Explores 
Evaluation Trends, Goals 



By ROY GABRIEL 

Northxpest Regional Educational Laboratory 

Drug program evaluation must be ''demys- 
tified'* if it is to be a powerful tool in prevention 
efifbrts, condxided members of a Western Center 
evaluation task force at a recent meeting* 

In September, Western Center evaluation staff 
met with national consultants to discuss future 
directions in the evaluation of local Drug-Free 
Schools and Communities programs. The group 
was charged with peviewir^ the Western Center's 
evaluationactivitiestodate and looking toward the 
next four years in prescribing evaluation assistance 
most needed* 

Invited participants included: Bonnie Beiuu-d, 
author of several research articles on prevention 
education, formerly with the Illinois Department 
of Education and currendy on a consulting ap- 
pointment with the Western Center's Far West 
Laboratory office; Jim Emshoff of Georgia State 
University, f ormeriy with the Office of Substance 
Abuse Education arid evaluation consultant to the 
Parents' Institute for Drug Education (PRIDE); Eric 
Goplerud, evaluation advisor to the Office of Drug 
Control Policy Office in Washington, D.C.; Tracy 
Manger, evaluation specialist with David 
Hawkins' Social Development Research Group at 
the University of Washington; and Judy Thome, 
project director of Research Triangle Institute's na- 
tional evaluation of the Drug-Free Schools and 
Communities prograixL 

Their recommendations can be summarized in 
four areas: 

1. Evaluation must be demystified if it is to be a 
useful component of a local program. There are no 
magic evaluation methods which are easily imple- 
mented at the local level. The figure in the next 
colimm illustrates the necessary linkages from 



goals to objectives to programactivities to intended 
outcomes. Process and outcome evaluation focus 
on the last two links in this chain. If local program 
staff can articulate these linkages, evaluation staff 
can help with the medtods to assess the efiective- 
ness of die program. 

2. Clear, concretciUxistrations of the ttsefulness of 
aw/uoiionresultsmtist drive WestemCenter efforts 
intrainix^andtechrucal assistance in this difficult 
area. When an evaluation plan dearly follows the 
program plan, as depicted in the figure above, its 
results provide useful feedback in refining the ac- 
tivities of the program. 

For cxanqjle, a process evaluation activity look- 
ing at tite implementation of the prevention cur- 
riailum may find that classroom teachers are 
devoting much more time to some units than 
others. Interviews with these teachers indicate that 
the material in many units is imfamiliar to them and 
the teacher's guide does not help. One solution 
may be to schedule a more structured training in 
the use of the curricultun. Another may be to or- 
ganize a regular but informal discussion group of 
teachers for the purpose of sharing how each of 
them implement the lesson plans. Bolstering the 
curriculum in this way not only improves its im- 
plementation, it improves the program's chances of 
realizing its intended outcomes. 

3. Identifying xnable, qmlity methods and instru- 
ments to measureprogram outcomes is a concrete piece 
of technical assistance sorely needed. The intended 
outcomes of local programs vary far and wide from 
the "bottom line" of reducing alcohol and other 
drug use. Objectives may include enhancing 
students' self -esteem, their feelings of optimism for 
the futxire, their sense of control over their own 
destiny, their attachment to the community around 
them, and so on. Assessing these affective out- 
comes can be tricky, and it is critical to identify 
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those who have the necessary technical charac- 
teristics to provide quality infonnation. Outcomes 
beyond the sdiool or district level can also provide 
infonnation on the more systemic effects of 
local progTMns. Trends in sudi commimity in- 
dicators as arrests for drug-related offenses, DWI 
citations^ and alcohol- or drug-related hospital ad- 
missions provide a broader-based look at what the 
sdiool and community are acoom|Hishing together* 
(See the artide "^tatiiJtical Profiler o n Page 11.) 

4. Cbntinued efforts at nrtiporttrtgt^ 
who are actwdy tvabioting thdr progrms will im- 
prove the state of the art of evaluation practice 
across the r^n. Among the 1^ school teams 
theWestemCinter has worked withinitsfirstthiee 
yearS/ there are a number of solid, infonnative 
evaluation plans imderway* The Center has iden- 
tifieda number of these and provided some techni- 
cal assistance throu^ its ^model evaluation site'' 
project 

Evaluation efforts in these sites focus on a 
variety of program conqx>nents: prevention cur- 
ricula, student assistance programs (see the artide 



on the SAP Assistant computer program bdow) 
and parent involvement to name a few. 

The experiences and advice of these local pro- 
gram coordinators and staff is invaluable toward 
meaningful evaluation practice. The Western Cen- 
ter will be organizing a conference featuring these 
individuals and their evaluation experiences. 
Fostering thdr communication with each other, 
and witii their peers across the region, will be a 
priority for the Center in the next few years. 

Wittt this guidance and reflection on our efforts 
over the past three years, the Western C^ter will 
continue to blend workshop tnuning, tailored tech- 
nical assistance, and product devdopment toward 
the meaningful application of evaluation in local 
programs. 

For more information, contact Roy Gabrid, As- 
sociate Director for the Western Center for Drug- 
Free Schools and Communities. 



From Western Center News, Volume 4, No. 1, December 1990 



1-21 



EVALUATION 

At its best: 

Informs alcohol & drug coordinators about what is working 
Supports funding decisions 
Reports to interested groups 
At its worst: 

Harasses program staff 
Provides useless information 

Even at its best, the evaluation effort may be viewed at times as a burden by program 
workers. Demonstrating that evaluation ultimately helps improve the quality and 
effectiveness of program services is the best approach to gaining the participation and 
cooperation of program staff. 
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SECTION 2: EVALUATION PRACTICE 



The next group of exhibits and readings is about evaluation practice. Perhaps the most 
basic task in this domain is to determine the feasibility of evaluating a particular 
program. The exhibit titled "Preconditions for Evaluation" provides an approach to 
making this determination. In this formulation, the "program" is carried out through 
"activities" which have a clear relationship to goals and intended effects. There is a 
definite rationale which connects the pro^am's activities to the outcomes being sought. 

The remainder of this section discusses various methods of data collection that can be 
used to assess both program implementation (formative evaluation), and program effects 
(summative evaluation). 



^ Q Western Regional Center for Drug-Free Schools and Communities 

2-1 . . 



ERIC 



i>U 



Section 2 



Evaluation Resource Notebook 



PRECONDITIONS FOR EVALUATION 



PROGRAM 



Activity 1 
Activity 2 
Activity 3 



Goal 1 



Activity 4 
Activity 5 



^ Goal 2 



A clearly articulated program 
Clearly specified goals and/or effects 
Rationale linking program to goals and/or effects 
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EVALUATION: THE BASIC STEPS 



Goals & Outcomes 
Indicators 
Data collection methods 
Comparison groups 
Analysis techniques 
Reporting strategies 
Use 



Evaluation: The Basic Steps 



Goals & Outcomes 

What is the intended overall effect of the program? What changes in individuals, groups, 
or systems are the intended result? 



Indicators 

If the intended result occurs, what are some specific observable changes or events that 
would also occur? Note that an indicator is not the same thing as a goal. An indicator 
provides a visible indication of the outcome of a program. 



Data Collection Methods 

Selection of indicators leads directly to planning a systematic method of gathering 
information about the indicators. Many alternatives may be available. Choice of a 
method can hinge upon (1) consideration of the general advantages and potential pitfalls 
of various methods, and (2) unique characteristics of time and place which may influence 
a program's implementation and outcome. 



Comparison Groups 

Some reference point will probably be needed in order to interpret the data collected 
about the indicators. Since indicators are usually characteristics measured within a group 
of individuals exposed to the program, it will also be important to make similar 
observations in a similar group of individuals who were not exposed. Solving practical 
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difficulties relating to establishing a suitable comparison group is likely to be a 
significant issue in the development of an evaluation design. 



Analysis Techniques 

Like the rest of the design, the approach to analysis should be specified before the project 
begins. This way, the evaluator can assure that the necessary information is collected- 
and that all of the information collected will be of value in the analysis. Most projects 
will require at least a descriptive data analysis. Often, the full potential of the findings 
will not be realized without the application of somewhat more specialized and advanced 
techniques for data analysis. 



Reporting Strategies 

No single report format or method of presentation will suffice for all the audiences and 
types of use the evaluation is likely to face. An understanding of the information needs 
of a given group or individual at a given time in the life of the program will be necessary 
in order to facilitate appropriate understanding and use of evaluation results. 



Use of Evaluation Findings 

Evaluation results can be used throughout the life of a program— while it is underway to 
guide mid-course corrections, if needed, and at the conclusion to contribute to the 
improvement of program design for the next generation of the implementation/evaluation 
cycle. 
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PLANNING ISSUES-UP FRONT 

Money (!) 

Recordkeeping 

Data Analysis Resources 

Time constraints 

Staff availability 

Money (!) 



Some of the factors to consider in planning an evaluation are listed in the exhibit titled 
"Planning Issues-Up Front." First and last on the list is "money (!)," which does not 
mean evaluation is prohibitively expensive, but does mean that evaluation costs need to 
be considered in plsjining. Some authorities have suggested that evaluation costs can be 
very minimal, because the needed expertise can be found among individuals already 
working in a program in other capacities, and other needed resources can be found at 
little or no cost. While such a happy circumstance may at times occur, it is more often 
likely that evaluation, like other services of value, will require some funding. 
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Advantages and Disadvantages of Various Data 
Collection Methods 



Questionnaires: 



Advantages: Easy to administer. 

Easy to tabulate responses. 
Answers remain anonymous. 



Disadvantages: No opportunity to clarify questions or responses. 
Requires respondents to read and write. 
Easy to fake responses. 



Interviews: 



Observations: 



Advantages: Questions can be clarified when needed. 

Unclear responses can be pursued. 

Does not require respondents to read and write. 

Disadvantages: Time-consuming to administer. 

Time-consuming to summarize responses. 
Requires training of interviewers. 



Advantages: Provides accounts of actual occurrence. 

Does not rely on respondent's recall. 
Does not rely on responses of respondents. 

Disadvantages: Intrusive. 

Requires U"aining of observers. 
Errors in coding may occur. 



Existing Records: 



Advantages: Wide variety of information is readily available. 
Relatively low cost to obtain. 
Does not require much training to collect 

Disadvaiitages: Records may be incomplete. 

Legal requirements may arise. 

May require time and effort to extract information 

needed. 
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Development of Questionnaire and Interview Instruments 



ERIC 



Planning 



Specific questions used in self-report instruments should be prepared in the context of a total 
evaluation plan. Many questionnaires and interviews yield useless information, simply because 
the cvaluator did not take the time to construct a plan. The following steps are suggested in 
planning your self-report instruments. 

1. Determine what information is to be sought. 

2. Determine what format the instrument will take. 

3. Develop the first draft. 

4. Revise the first draft. 

5. Pilot-test the instrument and revise if necessary. 

6. Write the detailed procedures for use. 

Question Development 

1. Is the question necessary? 

2. Is more than one question necessary to get the information you need? 

3. Do respondents have enough knowledge about the question to be able to answer? 

4. Is the question biased? 

5. Will the content of the question offend, embarrass, or otherwise lead the respondent to 
not want to answer? 

6. Is the wording clear? 

7. Is the language level appropriate? 

8. Do the words evoke an emotional response? 
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Sample Interview Questions 

A* For District/School Site Program Administrators: 

1. Describe the implementation of the activities, management procedures, and 
budget allocations of the program. 

2. How have you departed from your intended plan, if at all, in these three areas? 
Why have you departed? 

3. What particular problems have you encountered? 

B. For Teachers: 

1. How were you informed of the program and your role in implementing it? 

2. Describe your role in the implementation of the program. 

3. Did you receive training and/or assistance in carrying out your role? If so, please 
describe. Was it adequate? 

4. Did the program operate smoothly and efficiently in the classroom? If not, why 
not? 

5. Was the program well-managed? If not, why not? 

C. For Students: 

1. Describe the program to me. Tell me about the topics that were covered and how 
the material was presented to you. 

2. Did your teacher seem comfortable and well prepared in presenting the material? 

3. Were there any incidents that occurred while the program was being carried out 
thai stand out in your mind.? 

4. Did you enjoy the program? Was it intciestn-.g.' Did you learn from it? 

D. For Parents: 

1 . Are you aware of the program which is being carried out in the district/school? If 
so, how did you find out about it? 

2. Were you involved in the planning or carrying out of the program? If so, in what 
way? 

3. Has your daughter/son commented on the program? If so, describe comments. 

4. What is your opinion of the program? 
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Tips on Interviewing 

1. Ask friendly questions to help the person being interviewed feel more at ease. 

2. Restate questions when the interviewee looks puzzled. 

3. Express interest, periodically. 

4. Incorporate/repeat interviewee*s key phrases and terms in questions and in notes. 

5. Express/reaffirm complexity of program requirements. 

6. Avoid a series of questions beginning with 'why' (may contain a judgmental component). 

7. Ask for a concrete example if response is vague (I wonder if you could give me an 
example of that?). 

8. When the interviewee looks uncomfortable talking about a particular topic, "tread 
gently." 

9. Emphasize the sharing nature of the interview and. avoid judgmental queries. 

10. Acknowledge any "ventilation" of feelings that occur before proceeding to next question. 

11. If possible, try to take only brief notes during the interview itself and concentrate on the 
responses. 

12. Confirm/double check important points. 

13. Take a break during lengthy interviews. 

14. Jot down exact quotes or phrases once in a while. They add some personal flavor to 
notes. 

15. Allow only limited freedom for interviewee to go off on a **tangent." Maintain focus on 
improvement issues, rather than respondent (or your own) interests. 

16. At close of interview, allow for possibility to return to ask for any necessary clarification. 

Source: Spradley, J.S. (1979). The Ethnographic Interview. New York. Holt, Rinehart and 
Winston 



Western Regional Center for Drug-Free Schools and Communities 
Co 2-9 . , 

ERIC 



Section 2 Evaluation Resource Notebook 

Conducting Observations 

1. Classroom observations are a very expensive and time-consuming approach to 
evaluation, so be su'"e that you can afford and/or really need this information. 

2. The determination of what behaviors, events, and environmental features you will be 
observing is essential before deciding to use this measure. 

3. The kind of training needed for observers must be carefully planned. 

4. All observers need training. 

5. Observers should be carefully selected. You need observers that are unobtrusive, 
preferably not familiar with teachers or students to be observed. 

6. All teachers are not amenable to having an observer in their classrooms. A meeting with 
teachers to explain evaluation and alleviate fears is recommended. If possible, teachers 
should volunteer for this activity. 

7. Be prepared for complications in scheduling, e.g„ substitute teachers, shortened classes, 
school events preempting classes, students leaving on field trips. 
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DOCUMENT REVIEW CHECKLIST 



Deflnition 



A "document" may be defined as any kind cf written or printed information that can be used in 
support of or as evidence for a particular study. By collecting, sequencing, and interpreting 
# documents, the evaluator can trace the origins of a given situation and put it in context. 



Types of Documents 



Documentary information can include: 

• school records 

• census figures 

• personnel files 

• socioeconomic facts 

• test scores 

H • medical information 

• description of proposed program objectives, implementation plans, management 
plans, budget 

• demographic information 

• program philosophy 

• background material describing problem, content, and context 

• needs assessment information 

• information about community, school, industry 

• materials, such as objectives, lesson plans, instructional guides, teachers' manuals, 
textbooks, handouts, testing instruments 

• progress reports, evaluation reports 



Planning for Gathering Documentary Information 

1. What are the documents that are relevant to the program being evaluated? 

2. How do you determine the "relevant" documents? 

3. Whom would you ask to make this determination? 

4. Where are the locations at which these documents are stored? What are the hours these 
offices/locations are open for you to visit? Who knows this information? 

5. What are the names of the people who supervise/monitor/store these documents? Again, 
who knows this information? 

6. Whose permission do you need to examine these documents? 

7. Can the documents be removed from any of the locations or must they be reviewed at the 
site? Who can tell you this information? How long can you keep the documents? 

8. For each document or type of document what information are you looking for? 
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Access to Data Sources 

After determining the information you want to obtain, identifying the groups of people from 
whom you will be soliciting this information and the types of documents you will be reviewing, 
and matching with appropriate data collection instruments, then you will need to determine how 
to get to these people and documents for the information. In order to make this determination 
you will need to be informed of and act according to the following policies and procedures: 

1. District/school policies for conducting an evaluation and established procedures for 
contact with board members, staff, students, parent, and community school-based 
organizations 

2. The legal restrictions and requirements for examining and reporting confidential 
information 

3. Any school/community protocol to be observed in the conducting of an evaluation. 
This information is essential to know in the early planning stage of your evaluation so that: 

1. You discover if you can implement your evaluation 

2. You can revise any changes needed in the evaluation design based on school/community 
guidelines/policies/restrictions 

3. You can build in enough time to carry out needed access tasks. 
Some examples of access issues are: 

1, If you plan to interview students: 

Who needs to know and/or from whom do you request pvMinission? leachers? 
principals? superintendent? parents? 

2. If you plan to examine student records: 

What are the district, the federal, and the state policies regarding access to, and reporting 
of, confidential infomiation? 

Also, be aware of courtesies to be extended and attention to public relations in conducting an 
evaluation. Some people do not need to know about the study but your evaluation efforts would 
be well served if you provided this information to not only key members of the business 
community, but also to religious, civic, and other community organizations. 
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Student Assistance Programs 
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« student Records Policies and Procedures 



A school district, at a minimum, must adopt a written 
policy in compliance with the Family Education Rights 
and Privacy Act (FERPA), if it receives any funds from 
the U.S- Department of Education. Districts are strongly 
encouraged to go beyond this minimum requirement and 
adopt a policy and procedures which take into consider- 
ation other federal laws and regulations and state legal 
requirements. At the same time, schools will want to 
keep requirements for the maintenance of student 
records as simple as possible, and their use efficient and 
convenient for legitimate and valuable educational 
purposes. 

The following ^'model" is provided to assist district 
personnel in developing student records policies and 
procedures. 

Section One: Definitions 

For purposes of this policy, the district may wish to 
define such terms as: 

• Student 

• Eligible Student 

• Parent and Guardian 

• Education Records 

Section Two: Notification 

A provision providing for annual notification of 
parents of their rights is required. The method of 
notification must consider parents who have a primary or 
home language other than English. The method of 
notification must be specified, such as publication in the 
student handbook or direct mailing to homes. 

Section Three: Collection of Information From 
Students 

The programs and information should be specified 
which are subject to the requirement of advance parental 
consent before collection, as well as the method of 
obtaining consent. 

Section Four: Inspection of Education Records 

Parents or eligible students must have the opportunity 
to inspect and review a student's education records upon 
request. Optional methods for doing so may be speci- 
fied. 

The parents or eligible students should submit a 
request to a specified school official, identifying as 
precisely as possible the record requested. 

The school official needs to arrange access promptly 
and notify the parent or eligible student when and where 
the records may be inspected. Access must be given 
within 45 days of the request 

A parent or eligible student may not inspect or 
review the portion of a record pertaining to other stu- 
dents. 

Section Five: Inspection of Student Assistance 
Program Records 

Special provisions should be spelled out related to 
confidentiality of alcohol and other drug use records. 



Section Six Inspection of Instruction Materials 

Programs and materials should be specified which 
are subject to the request of parent inspection, as well as 
the procedure for responding. 

Section Seven: Refusal to Provide Copies 

Although the school may not refuse a request to 
inspect and review a record, it may specify circum- 
stances where a copy of the record will not be provided 
to parents; for example, if the record includes answers 
to a standardized test 

Section Eight: Fees for Copies of Records 

The fee for copies must be specified, but the actual 
fee is optional. The district may not charge for search 
and retrieval of records; it may charge for copying time 
and jpostage. 

Section Nine: Type, Locations and Custodians 
of Education Records 

A list should be provided of the types of records 
maintained by the district^ such as: 

• Cumulative School Records 

• Health Records 

• Speech Therapy Records 

• Psychological Records 

• Student Assistance Program Records 

• School Transportation Records 

• Test Records 

The location of each and the custodian (school 
principal, health director, psychologist, pupil transporta- 
tion director, etc.) also should be listed. 

Section Ten: Disclosure of Education Records 

A provision is required specifying exceptions to 
confidentiality; that is, under what circumstances, and to 
what individuals and agencies, will information be 
disclosed without parental consent 

Section Eleven: Records of Request for 
Disclosure 

Provision must be included for maintaining a record 
of requests for infonnation disclosure, including the 
individual or agency making the request, what informa- 
tion was requested, and whether the request was 
granted. 

Section Twelve: Directory Information 

If the district decides to exercise the option of 
disclosing directory information, items which are to be 
made available should be specified. 

Section Thirteen: Correction of Education 
Records 

Provision must be made for the correction of records 
Procedures should cover the method of requesting, 
options for the district in responding, and metliods for 
arranging and conducting a hearing, issuing a written 
decision, and amending records. 
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Frequently Asked Questions 



Q: What kind of notification needs to be provided to 
parents and when ? 

A: A district must notify parents annually of the 
requirements under FERPA. If parents have a 
primary home language other than English, the 
district must effectively notify them. Notifica- 
tion typically is done by publication in the 
student handbook or at die beginning of the 
school year by mail. 

Q: What reco rds are subject to confidentiality and 
consent requirements? 

A: Any record, in handwriting, print, tape, fdm, or 
other medium, maintained by the school or an 
agent of the school is covered except: 

• A personal record made by an indi- 
vidual school staff member and kept in 
his or her personal possession, made 
available to no one other than the 
person's temporary substitute 

• An employment record used only in 
relation to a student's employment by the 
district 

• Alumni records containing informa- 
tion about a student after no longer 
attending the district 

Q: Is parental consent required to conduct a sur- 
vey of students* attitudes about and use of 
alcohol and other drugs to obtain information 
for planning a school prevention program, if 
individual students are not identified? 

A: The only federal restrictions on gathering infor- 
mation from students are provisions of the so- 
called Hatch Amendment. These provisions 
only apply if funds provided by the U.S. Depart- 
ment of Education are used to conduct the 
survey, and not if state and local sources of 
funds are used. If parental consent is required 
before surveying students, the requirement ap- 
plies regardless of how the information is to be 
• used or reported. 

Q: Should certain records, such as those related to 
student alcohol and other drug use assistance 
programs, be kept separate from other school 
records, such as genet al academic records? 

A: Probably yes. which may furtlier complicate 
student recordkeeping systems and procedures. 
Different confidentiality and coasent require- 
ments may apply to various alcoliol and other 



drug use records maintained on students. The 
result may be three or more different "catego- 
ries," depending on the combination of federal 
and state requirements which apply. 

Q: What information can be released to an indi- 
vidual, the press, or the public in general 
without prior parental consent? 

A: Information which is not "personally identifi- 
able" can be released. Typically, this is aggre- 
gated data in the form of program evaluation 
reports and schoolwide or subjectwide achieve- 
ment reports. Likewise, "directory infonma- 
Uon" may be released including names, ad- 
dresses, telephone numbers, major field of 
study, date and ^lace of birth, participation in 
activities and sports, dates of attendance, de- 
grees and awards received, most recent school 
previously attended, and photograph. How- 
ever, names and other infonnation about stu- 
dents participating in any alcohol and other 
drug use prevention, diagnosis, referral, or 
treatment activity are confidential and may not 
be disclosed without prior written consent 

Q: Are there certain individuals or organizations 
that confidential information can be provided 
to without parental consent? 

A: Some of the specific parties to whom the law 
allows schools to disclose otherwise confiden- 
tial student information are: 

• School employees who have a "need 
to know" - Other schools to which a 
student is transferring 

• Certain government officials carrying 
out their functions 

• Appropriate parties in connection 
with financial aid to a student 

• Organizations doing studies for the 
school 

• Accrediting agencies 

• Persons who need to know in case of 
health and safety measures 

• A school must keep a record of any 
such request and disclosure. 



Western) Regional Center for Drug-Free Schools and Communities 

2-16 



f 

• Qua/ Requirements: Confidentiality and Consent 



The dual requirements of confidentiality and 
consent arc closely allied issues which school dis- 
tricts face in maintaining student records necessary 
for the efficient and effective operation of their 
educational programs. A complicated set of federal 
and state laws and regulations apply. Some apply to 
most student records regardless of the source of funds 
supporting the program, the educational subject, 
whether it is part of the core curriculum or an experi- 
mental program, or the purpose for which information 
is gathered and used. Other laws and regulations 
apply specifically to alcohol and other drug use 
programs and activities, or specifically to experimen- 
tal programs, or only to federally funded activities. 

First, every school district should develop, adopt, 
and implement a clearly stated student record policy 
and procedures. 

Second, the staff, parents, and students should be 
informed about the policy and procedures, so that 
they understand requirements, their rights to access, 
and restrictions on such rights. 

Third, in implementing a records system, district 
staff should examine carefully each set of laws and 
regulations to determine what student records are 
subject to them. 

This guide was prepared to provide school dis- 
tricts with basic information for planning how to 
proceed in completing these tasks. Information is 
provided about the three primary federal require- 
ments. Most states also have applicable laws with 
varying requirements. 

Because the topic is legally complex, school 
districts are advised to seek legal counsel on issues of 
confidentiality and consent prior to developing a 
policy and procedures. 

Applicable Federal Laws and Regulations 

Requirements and restrictions on student records 
related to drug and alcohol and other drug use preven- 
tion and intervention activities are spelled out in three 
major federal laws and regulations. 

1. The Family Educational Rights and Privacy Act 
(FERPA) requires that educational agencies provide 
information contained in student records to students 
who are 18 and parents of students who are not yet 
18. Further, it precludes schools from disclosing this 
information to others, with certain exceptions. 

2. Student Rights in Research, Experimental Activi- 
ties, and Testing (the Hatch Amendment to the 
General Education Provisions Act) requires parental 
consent for a student to participate in programs 



involving psychiatric or psychological testing or 
treatment, or designed to reveal information pertaining 
to personal beliefs, behavior, or famUy relationships. 
It also gives parents the right to inspect instructional 
materials used in research or experimentation projects. 

3. Confidentiality of Alcohol and Drug Abuse Patient 
Records regulations issued by the U.S. Department of 
Health and Human Services also apply to school-based 
programs, providing for confidentiality. 

Family Educational Rights and Privacy Act 

The Family Educational Rights and Privacy Act 
regulations became effective in 1976. Basically, the 
law says federal funds may be withdrawn if an educa- 
tional agency fails to provide parents or legal guard- 
ians access to their child's educational records. It also 
precludes schools from disclosing this infomiation to 
others without the consent of parents or guardians. 
After students reach the age of 18. they may exercise 
these rights on their own. 

There are few exceptions to the requirement for 
prior consent before releasing infomiation, usually 
requiring a court order or overriding state law. 

If a parent, guardian, or student over age 18 reviews 
the information and believes it is misleading, inaccu- 
rate, or violates a student's protected rights, the 
information can be amended. A hearing may be held if 
there is disagreement. 

In virtually all cases, the student assistance program 
records maintained by a school district are subject to 
FERPA requirements. 

Student Rights in Research, Experimental 
Activities, and Testing (Hatch Amendment) 

The General Education Provisions Act requires that 
instructional material in federally assisted research, or 
experimentation projects designed to explore new or 
unproven teaching methods or techniques, be available 
to the parents of participating students. Furthermore, 
no student can be required to participate if a parent 
submits a written objection. 

The Hatch Amendment, passed in 1978 and regu- 
lated by the U.S. Department of Education since 1984, 
further requires parental consent before the student 
participates in programs involving psychiatric or 
psychological examination, testing, or treatment 
designed to reveal infomiation f)ertaining to personal 
beliefs, behavior, or famUy relationships. 

The regulations are sweeping in that they define 
psychiatric or psychological examination or treatment 
as including activities that arc not directly related to 
academic instruction and are designed to obtain 
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personal infonnation, behavior, or attitudes. 

They apply only to activities supported by funds 
provided by the U.S. Department of Education, not 
to all school activities. 

Confidentiality of Alcohol and Drug Abuse 
Patient Records 

These U.S. Department of Health and Human 
Services regulations, as amended in 1987, clearly 
apply to school-based programs that deal with the 
referral of students for treatment for alcohol and 
other drug use. While the regulations apply to 
**federally assisted programs," this is generally 
assumed to include any organization receiving any 
federal assistance (including state pass-through 
fimds). 

^Tiile school programs rarely diagnose or label 
students as alcohol or drug dependent, they do 
**refer'* students who display certain signs and 
symptoms which may be characteristic of alcohol 
and other drug dependency to assessment. While 
one could argue the school has made no such diag- 
nosis or labeling of alcohol or divg dependency, the 
mere fact of referring, based on certain signs and 
symptoms associated with dependency, could be 
considered as referring alcohol and drug dependent 
students. 

In general, these regulations prohibit information 
being supplied to anyone about persons in an alcohol 
or drug related program, unless the student and 
parent consent; there is a court order; disclosure is 
made to medical personnel in an emergency: or tlie 
information is used for research, program evaluation, 
or audit purpose. 

Collection of Student Information 

The only restriction on iJic collection of infonna- 
liou from students is a provision of the Hatch 
Amendment requiring consent of an adult or emanci - 
patcd student, or the parent or guardian of a minor 
student. This provision only applies to federally 
funded activities which are a part of a research or 
development project. 

Release of Student Information 

In general, prior consent of a student, parent, or 
both is required before a school can release **pcrson- 
ally identifiable infonnation"; tliat is, when it is 
information about a specific, idcnlifiecl student. 

Infomiation which is not "personally identifiable'' 
generally can be used for such purposes as research. 



program plamiing and program evaluation, and 
reports of the results may be made available to the 
public. 

Each of the three federal laws and regulations has 
specific exceptions which permit schools to release 
otherwise confidential information without student or 
parent consent to certain persons and agencies for 
certain purposes. Examples include release of 
information to school employees, including teachers, 
who have a legitimate educational interest in the 
student, and transfer of records to another school 
where the student seeks to erux)ll. Specific provisions 
should be looked at carefully in these instances. 



The Western Regional Center for Drug-Free 
Schools and Communities is one of five regional 
centers in a national network established by the 
US. Department of Education through the Drug- 
Free Schools and Communities Act of 1986. The 
Center serves the following states and territories: 
Alaska, American Samoa, California, Guam, 
Hawaii, Idaho, Montana, Northern Mariana 
Islands, Nevada, Oregon, Patau, Washington, 
Wyoming. 

Western Regional Center for Drug-Free 
Schools and Communities 

Judith A. Johnson, Director 

Northwest Regional Educational Laboratory 
101 S.W. Main Street, Suite 500 
Portland, OR 97204 
(503) 275-9500 or (800) 547-6339 

Field Office 

1164 Bishop Street, Suite 1409 
Honolulu, Hawaii 96813 
(808) 532-1904 

Far West Laboratory for Educational Research & 

Development 

730 Harrison Street 

San Francisco, CA 94107 

(415)565-3000 

Southwest Regional Laboratory 
4665 Lampson Avenue 
Los Alamitos, CA 90720 
(310) 598-7661 
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SECTION 3: NEEDS ASSESSMENT 



When information is needed about the nature and scope of a problem, a needs assessment 
may be conducted. Problems typically become more visible and better understood as the 
available information becomes more complete. Early reports conceming a problem tend to 
be fragmentary and anecdotal. If such information seems to indicate there is cause for 
concern, oftentimes pressure will build within institutional systems to "do something" about 
the problem. Needs assessment can thus be viewed as an important early step in the process 
of fashioning an effective and appropriate response to the problem. 

The purpose of needs assessment is to gather information which is more data-based and 
comprehensive, and collected systematically, than anecdotes or hearsays. The purpose is net 
to fish for alarming statistics intended to support a pre-selected course of action, nor is it to 
identify problem solutions or strategies (although information derived from needs assessment 
may of course be used in this process). 

Information about the extent of a community's alcohol and other drug problems can be used 
in three major ways: 

1. To establish a common understanding of the nature and extent of problems in the 
community. 

2. To overcome the common human tendency to minimize or deny the existence of 
unwelcome information. 

3. To establish a baseline, before intervention, to use as a comparison point for a later 
assessment of program effectiveness. 

Needs assessment can also help suggest priorities for the program implementation. For 
example, services can be targeted toward particular age groups, current programs or activities 
can be reinforced or refocused, or other available resources can be identified. 
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Needs Assessment: 
Process and Strategies 



BY CAROL F. THOMAS 
Southwest Rtgioml Laboratory 

What is the extent of alcohol and drug 
use by students in our school and adults in our 
conununity? What is the pattern of abuse 
locally? What factors are associated with use? 
What resources exist in our school and 
comnwmity for solving the alcohol and drug 
abuse problent? Inxplenaenting an ongoing 
needs assessment process in your conununity 
can provide the answers to these and other 
important questions useful in the prevention of 
drug and alcohol abuse. 

Needs assessment is a label applied to a 
large number of data gatherings analysis, and 
interpretive activities conducted for the purpose 
of providing direction for, or jiistifying, 
programs and policies. The process of needs 
assessment has acceptance as a common sense 
notion because program sponsors and program 
directors want a more objective and rational 
basis for nrtaking resource allocation decisions. 

In the area of substance abuse, there are 
many compelling reasons for engaging in the 
needs assessment process. Over the last five 
years, more and more school districts and 
conummities have begun to implement 
prevention programs to address the problem of 
substance abuse. If these efforts are to be 
effective, program planning must proceed based 
on a careful analysis of the problem, and 
continuous monitoring of the process. 

The results of a comprehensive needs 
assessment can be used to guide the planning, 
development, and implementation of services 
and programs to fight substance abuse. 

There are many activities being lumped 
together under the label "needs assessment" that 
can have quite different purposes. Needs 
assessment can direct the planning process, 
determine the extent and nature of problems, 
identify specific problems to address, assist in 
the development of a program philosophy, 
determine resources available, identify needs for 
technical assistance from others, and motivate 
people to action. Needs assessment is not a 
fishing trip for any and all kinds of data, ihe 



focus should be on the need for drug and 
alcohol prevention and intervention programs in 
the schools and community. The data should be 
collected with a purpose, use, and audience in 
nund. 

Needs Assessment; Program Planning 
and Evaluation 

It is important to remember that needs 
assessment is only one component in a 
comprehensive program for drug and alcohol 
abuse prevention. The platming process is 
another way of deciding: Where are we now? 
Where are we going? How are we going to get 
there? And how will we know when we have 
arrived? 

These questions illustrate the 
relationship between heeds assessment, 
program planning and program evaluation. We 
ask "Where are we now?* to identify needs, 
examine data, and determine strengths and 
weaknesses in present programs. The question, 
•*Where are we going?" relates to our goals that 
we want to achieve. "How are we going to get 
there?" is the steps or methods we choose to 
reach the goals. 

The last, "How will we know when we 
have currived?" relates to evaluating our efforts. 
It is how we can make mid-course corrections to 
our program implementation and tell others 
whether our efforts have l>een successful. 

Why Do a Needs Assessment? 

Our mission is clear-to promote drug- 
free schools and communities. Needs 
assessment identifies, as closely as possible, the 
problems that the planning process will address. 
As a precursor to the planning process, needs 
assessment enables long-and short-range goals, 
objectives, activities, and resources to be applied 
mere effectively to the problem. 

Generally the following types of 
problem-oriented data are collected: 1) school 
and community drug-use indicators that give 
data on the incidence and prevalence of drug 
use; 2) legal and problem behavior indicators 
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that are believed to be correlated with drug use, 
such as crime rates; 3) health indicators that 
provide information on drug-related 
emeigendes and use of treatment facilities; 4) 
p^chological or developmental characteristics 
that are believed to be correlated with future 
drug use, such as aspects of family interaction; 
and 5} social or ecoiu>mic conditions that 
correlate highly with drug use, such as 
persistent unemployment 

Another important reason to conduct a 
needs assessment in your community is to 
determine what resources are currently 
available to address the drug and alcohol abuse 
problent. Resources will include the number 
and types of programs that exist;, the services 
provided, the availability of the services to 
potential clients, and the finances, staff and 
materials available. 

This type of needs assessment, when 
compared with the resources needed to solve 
the problem, can identify duplication of effort or 
imderutilized resources. It can help set 
priorities for target groups, areas of i>articular 
concern, political considerations, and so forth. 

The discrepancy between the resources 
needed and the resources available will have an 
impact on the feasibility of any plan. Sometimes 
it also identifies the need for particular expertise 
which needs to be obtained from outside the 
particular agency. 

Steps Necessary for Conducting a 
Needs Assessment 

The first step in any needs assessment 
process is to identify the purpose of the needs 
assessment. The objective of this step, however, 
is to brainstorm with your key stakeholders and 
jot down the responses to two questions: Why 
do we need to do a needs assessment? What 
will we do with the information? Given this 
information, the next task is to review it and 
prioritize the various reasons your group might 
have for conducting a needs assessment. This 
step sets the stage for your needs assessment 
activities and motivates the group to participate 
in the process. 

Step two is to organize the effort and 
assign responsibilities so that the process is 
successfully carried out A key person should 
be identified who will be committed to seeing 
that tasks are accomplished. This does not mean 
that one individual alone will be able to handle 
the job. Individuals, groups, or agencies also 
should be identified that can work on 



subcommittees or task work groups, or can 
contribute resources to the process or who can 
cooperate in (he needs assessn^ent by providing 
data or information. In addition, individuals, 
groups, or agencies that will need to be 
ii\formed about the needs assessment results, 
implement the findings, or be affected by the 
results should be noted. This step is important 
to ensure that any fears about the needs 
assessment are allayed and that key persons 
have bought into the process and will facilitate 
the data collection activities. 

The third step in the needs assessment 
process is to establish the information 
requirements and identify existing data. The 
questions that need to be addressed are: What 
information do you need to document tiie 
problem? Where is or who has the information 
needed? Your needs assessn^nt task force 
should brainstorm to identify the sources of data 
in your community. What methods will 
generate relevant inforn\ation? How much new 
data collection will be needed and how much 
information can be obtained from existing data? 
If resources permit, it is advisable to use 
multiple methods to gather infomution about a 
problem. In this way you can have more 
confidence in the results. 

The fourth step is to develop the needs 
assessment plan. This plan should incorporate 
information obtained in the previous steps. In 
developing your action plan, consider the 
resources and constraints that may affect which 
needs assessment techniques you choose. For 
example, how much money is available to be 
applied to this effort; what other resources are 
available; who can supply in-kind services; how 
much time can be allocated to the assessment? 
Consider the various techniques available for 
collecting the information you need and decide 
which fit your resources and constraints. 

Step five is to implement the plan, 
document, and judge the results for Chose 
involved in program planning. A common 
understanding of the problem will be developed 
through the needs assessment process. Problem 
statements can be generated that sununarize the 
conclusions of various pieces of information. In 
addition, a clear picture of the resources that 
exist in your community will emerge. 

Reporting needs assessment results can 
help combat denial that may exist in the 
community because it can identify a pattern of 
drug and alcohol abuse locally. The needs 



assessment will also establish a baseline for later 
assessments of the effectiveness of various 
school and conunimity prevention or 
intervention strategies. 

Political Process 

Planning for substance abuse programs, 
whether in the schools or in the community, 
requires collaboration between a variety of 
people and agencies— collaboration in 
identifying the purpose for the needs 
assessment, for assistance in collecting data, and 
for support in discussing the results in the 
community. G>llaboration reflects and is 
embedded in an American tradition that 
assumes that groups of institutions, agencies, 
and conununity representatives are more 
effective in solving complex problems than if 
independent and unilateral actions are taken. 

The needs assessment process 
necessitates negotiating value systen\s among 
constituencies with different interests and 
objectives. G)llaboration requires people to 
extend themselves into unknown and less 
comfortable areas. Such time coivsuming efforts 
are seldom rewarded. Collaboration is 
facilitated when there are clear-cut, agreed-upon 
goals and functions. The effectiveness of any 
collaborative effort will depend on the abilities 
of the partners to accurately and honestly assess 
their capabilities and those of their partners and 
their ability to develop an atmosphere of trust so 
that no member of the collaboration feels that he 
or she is being preempted. 
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statistical Profile: 
A Good Place to Start 



■ The Regional Drug Initiative has 
assembled data on health, crime, 
and substance abuse as a focus for 
community action 

ByROY GABRIEL 

NorOrwestRegiofudEducatiorudLa^^ 

Cbmsmnities aooss the nation aie getting in- 
volved in the fig^ against substance uae «nd abuse. 
FMeral funding, tr&ung^ and technical assi^ 
are inoreasin^y supporting efforts which begin 
with and are sustained by tiie commitment and 
energy of local leader;^ 

A critical step in starting off on the right foot is 

the assembly of objective data from a variety of 
sectors to lookat the severity and complexity of the 
problem. Few would argue with the significance of 
this "'needs assesssment" step, but few coalitions 
know how to get started. ''Lef s do a survey" is 
usually not the best start There are informative 
data already being collected by public and private 
agendes. But it is not immediately obvious how to 
get at ^ information. 

The Regional Drug Initiative (RDD, a com- 
munity coalition in Portland, Oregon, has met this 
challenge. It has assembled data in health in- 
dicators (such as the number of drug-affected births 
each year), crime indicators (such as the number of 
adult arrests for drug offenses), incidence and 
prevalence of substance use (such as the percent of 
eighth-grade and 11 -grade students using 
marijuana in the last 30 days), and a ntmtber of 
other indicators. By putting these together and 
displaying them in an informational brochtue, the 
Regional Drug Iniiiative Drug Inxpact Index, the RDI 
has aroused public opinion anid focused its own 
commtmity action. 

Working collaboratively, the RDI and the 
Western Center used a six-step process: 

Step 1: "brainstorm'' a list of possible indicators 

Convene a focus group of coalition members to 
generate an initial list of indicators. A group of six 
to eight individuals, representing diverse con- 
stituencies in the community, such as law enforce- 



ment, education, and health care — is recom- 
mended. Aquestion like ^'Whatdatahave you seen 
that make you think we have an alcohol wd drug 
problem in this commuxu^?^ can get things rolling. 

Important At this stage, all ideas have equal 
weight and are not screoed using any sense of 
ri^ vs. wrong or important vs. trivial. 

Step 2z Round oat the field 

Once the ^wish list^ is established, one or two 
focus group members can begin to duster the in- 
dicators into categories or areas: health, crime, 
safety, treatment, etc This lends some structure to 
the list and can help identify information gaps, 
redundancies, and other problems. 

Step 3: Find available data 

Focus group members, or their associates, will 
likely know where statistics are kept in a number of 
these areas. Sources mig^t indude county or state 
healthdepartnwits, traffic safety commissions,and 
local police stations. Organizational and 
departmental structures vary from community to 
commxmity, so be prepared for a series of phone 
calls for each indicator. You will devdop your 
detective skills as you work your way through 
these systems. Keep track of your phone calls with 
some sort of tdephone log system. Generally, you 
will find one key person in an office whose job it is 
to put these data together each month, quarter, or 
year. 

Important: When you find this person, rejoice 
and celebrate! Also, nxirtxire the rdationship. You 
will need this individual's cooperation again and 
again. 

Step 4: Screen indicators in terms of their 
availability 

Returning to the focus group, members can 
report back how easy or difficult it is to obtain the 
information. Some data will simply not be avail- 
able. Most, however, will be there, but not in 
predsdy the form you want it (monthly rather than 
annual totals, percent rather than numbers, ail 
crimes rather than distinct categories, etc.). Don't 
be discouraged. With some additional help, you 
will likely be able to translate available data into the 
form you need. 
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Important There are voluxnes of data out there. 
If you find some interesting things you hadn't an- 
ticipated^ bring them back to the group, but don't 
get tidetradoeL Some of the best begiiuvings have 
been lost amid masses of printouts, reports, and 
files. 

Step 5: Look for state or national compansons 

Whileifstruethatyouarelookingforindicators 
that describe your own commtmity, one of the first 
questions your audience will raise is ^thaf s inter- 
esting, but how do we compare to other com- 
irunitiesr The detective worit you have already 
done will hdp you here. Typically, yoiu: key a)n- 
tacts who have supplied you with local data will 
know where they soul it where it is aggregated to 
a larger population base, etc 

Step 6: Select a 'final list^ of indicators 

Witti the extensive data gathering you've done, 
you are ready to finalize the list Use at least these 
criteria: technical quaiiQr (reliability aiui validity), 
variety (represent several different indicator 
categories for a truly complete picture), and acces- 
sibility (wei^ the ease or difficulty of obtaining 
these diUa, because you will likely be doii^ it again 
in the future). 

Important No indicator will be perfect Limita- 
tions in individual indicators will be ofiset by the 
strength of the complete set In this endeaver, the 
whole is dearly greater tiian the sum of its parts. 



Once you put your indicators together, 
strategies for dissemination must be devdoped. A 
simple presentation, using charts and graphs with 
minimal narrative, is recommended. The example 
above, rxi excerpt from the Rtgiond Drug Initiative 
Drug Impact Index brochure, shows the format 
devdoped by the Portland Regional Drug Initia- 
tive. Telegraph is accompanied by tiueeslKjrt nar- 
ratives: (coxmter-dockwise from upper left) the 
name of the indicator, a frru^ teduucal note to alert 
the reader U) any major anomalies in the collection 
of tlat data^ and interpretive remarli;s to represent 
the coalition's imderstuiding of ibe data* 

Once dte information has been disseounated, be 
ready. Heal data'^sudi as these provoke loud and 
varied reacdox\s« The investment of the focus 
group and the ovendl oommunitycoalition will pay 
huge dividends here. The indicators sdected were 
setocted for a reason. Their meaning and inter- 
pretation have already been discussed at length 
within die coalitioiu 

Cjet ready for actiord Armed with data and 
information on the array of effects of substance 
abuse,preventionplaxmingcan proceed with focus. 

The Regional Drug IrtUiative Drug Impact Index 
brochure (Jmie 1990) is available from the Western 
Center's Resource Onter. Also available is an in- 
structional handbook entitied. Developing A Com- 
mumtf Profile (September 1990). 
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Choosing A Student Use Survey 



By JACK POLLARD 
Southti^est Regional Laboratory 

School administrators often ask if there are 
very wany surveys available measuring student 
use of alcohol and other drugs. The answer is: 
Yes, there are a loti With the explosion of 
concern about student drug use, many survey 
instrximents have been developed that are 
appropriate for surveying students in a school 
setting. Some of the surveys are in the public 
domain. Some must be purchased, and at quite 
a price. Some sun^eys are generally very good, 
but'some need-as they say-further revisions. 

Because there are so many surveys 
available, choosing the best survey for your 
needs can become quite a headache. 
Consumers should be alert to several key issues 
before making the decision. Many issues are 
ot)vious, such as how much does the survey 
cost Other issues are not so obvious, such as 
whether the test is accompanied by detailed 
instructions concerning administration and 
scoring. 

Below are some of the more important 
issues in selecting a student use survey. Not all 
the issues have been addressed, indeed, it 
would be impossible since the choice of the best 
survey depends, in part, on the particular 
consklerations of individual districts or schools. 

Topics Covered: A good survey will 
assess students on three important dimensions: 
dmg use, dmg attitudes, and drug knowledge. 
Alcohol and marijuana, being the most 
commonly used drugs, are most often surveyed. 
Then there is a broad range of other illicit dmgs 
that you will want to consider: cocaine, crack, 
heroin/opiates, LSD, POP, sedatives, steroids, 
stinujlants. tobacco and tranquilizers. It is 
unlikely that any one survey will cover all of 
these dmgs-you must pick the survey that 
covers the topics that are of most interest to 
your school or community. 



Grade Levels: Sun^eys are written for 
specific age ranges/grade levels. Particulariy in 
the younger grades, the survey needs to be at 
an appropriate level. Read a survey for 
appropriateness before purchasing or adoption. 
Often, a surve/s promoters will say It is 
appropriate for an eariy grade (e.g., grade 4), 
but it will be too difficult for younger children to 
read or to understand the instructions. 

Cost and Support Services: Purchased 
surveys will cost from about 30 cents per sun^ey, 
up to as much as $2.00. That can get expensive 
if you have a lot of students you want to sun^ey. 
The range of support services can be equally 
varied, tf you adopt a public domain survey, you 
may or may not have accompanying instructions 
for administration or scoring. With a purchased 
survey, you're likely to receive more support, 
including possible machine scoring and a written 
report interpreting and summarizing the sun^ey 
responses. 

Other Features: Some of the other 
features that you may want to conskler: Are 
Spanish (or other language) translations 
available? Are there noons available on this 
test? How tong will this test take the average 
student to complete? And last but not least, is it 
copyrighted? 

With all of the sun^eys available, ft's really 
unnecessary to reinvent the wheel. Choosing a 
good survey is just a matter of putting the time 
into looking through whaVs available, and finding 
what's right for you. 

For a list of available surveys and tips for 
planning a student use survey, contact The 
Western Center office nearest you. 
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USING STUDENT USE SURVEYS 


1. 


Assess needs-planning 


2. 


Baseline information 


3. 


Heighten awareness of community to extent of student use. 


4. 


Evaluate program impact 



Student use surveys may be used at several stages of the planning and evaluation process 
for AOD prevention programs. As a measure of AOD use prevalence in a community or 
population, the survey plays a role in needs assessment. Information from such a survey 
may also increase community awareness, and reduce denial, about the extent of 
problematic use. Prior to beginning services, survey data can help establish the 
"baseline" AOD use level for those individuals who are to receive the service. 
("Baseline" refers to the situation before the services were provided. In advertisements 
which feature "before" and "after" photos of participants in weight reduction or body- 
building programs, the baseline is the "before" picture.) Once services are underway, and 
afterward, the student use survey can help evaluate outcome by showing the extent of 
change from baseline levels of use. 

An example of a short (15 questions) student use survey, with instructions for its 
administration and analysis of results is included in Section 7. Other instruments are 
available in the public domain or through commercial sources. See the Western Regional 
Center publication titled "Surveys of Student Alcohol and Other Drug Use: A 
Consumer's Guide," 2nd edition, for a review of 27 survey instruments. 
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Tips For Planning a Student Use Survey 
Compiled by Carol F, Thomas, Ph.D. 
Western Regional Center for Drug-Free Schools and Communities 
101 SW Main Street, Suite 500 
Portland, Oregon 97204 



Survey Elements Comments/Recommendations 

Clear Goals • Provide an information base for efforts to prevent alcohol and 

drug use. 

• Heighten awareness of legislators, officials, and the public 
about the need to improve prevention efforts. 

• Have a baseline for later evaluation of prevention efforts, 

• Identify clusters of schools to be used as normative reference 
points for similar schools who do not do a survey. 

• Get a profile of current prevention curricula, 

• Keep all abused substances in perspective. 



Prc-survey planning • 



Plan for the publicizing and use of survey results before the 
survey is given. 

Plan to ensure that no important decision-makers are left out 
(i.e„ school board, superintendent, informal gatekeepers, 
principals, teachers). 

Field test or review the content of the survey. The survey 
should be designed so that students who don't use drugs will 
take the same amount of time to complete the survey as there 
who do use drugs. 

Include student input in developing or planning the use of a 
survey. 

Evaluate the district's policy with respect to this kind of 
infomiation gathering process. 

Carefully review the survey for clarity, accuracy, readability, 
and utility before it is administered. 



Format • Include brief, simple instructions. 

• The survey should be attractive and easy to read. 

• Include clear items that are not confusing or subject to 
misinterpretation. 



Content 



Questions about student use should be behavioral, e.g., "How 
often do you drink beer?" rather than "Do you have a drinking 
problem?" 



ERJC 



Wastarn Ragional Center for Drvg-Fraa Schools and Communitias 

3-9 



Section 3 



Evaluation Resource Notebook 



The survey content should be coordinated across school 
buildings so that the responses by students at different grade 
levels can be compared. 

Phrase questions to avoid inadvertently casting drugs and 
alcohol in a favorable light. 

As much as possible, the survey should not suggest methods of 
using drugs, possible sources of, nor benefits from using drugs. 



Sampling 



• Random sampling can be used to reduce the number of 
students surveyed, but is difficult to implement in school 
settings. Consider administering the survey to selected grades 
instead. 

• Consider administration to at least one grade in each school 
building, including elementary schools. 



Relevance 
Comprehensiveness 



A survey's questions should be similar to those of the 
widely used state/national surveys for future comparison 
purposes. 

Survey content should be constantly updated to reflect the 
changing drug environment. 



Validity/Reliability 



Meticulous wording of the survey items is essential to ensure 
reliability and validity 



Exaggerated Drug Use 

• Not many students do it, according to researchers. 

• Exaggerators and minimizers cancel out effects in overall 
results. 

• A fake drug ca i be added to the survey. If a large number of 
students say they use the fake drug, you would suspect 
exaggeration on other drugs as well. 

• Eliminate ob% lous exaggerators from the results (e.g., students 
who say they use heroin but never marijuana. The results 
would be more conservative, and would underestimate use, a 
preferable alternative from the perspective of needs 
assessment. 



Random Responding by Students 

• There is a lot of evidence of consistency, not random 
responding. Percentages of use/nonuse for each substance vary 
in a logical way. 

• Ask about the substance in several different ways and run a 
"consistency check." If the student is inconsistent more than 
once, delete the survey. 
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• Use a statistical method: "internal consistency reliability 
index." 

• Compare trends in use of drugs and compare your survey 
results with the results of other surveys. 

Confidentiality/ • Guarantee the anonymity of subjects taking the survey. 
Consent 

• Survey students in an appropriate room other than their own 
classroom. 

• Adequate proctoring is recommended (without the proctor 
being able to observe responses made to specific questions by 
students). 

• Stress the importance of the survey and its voluntary and 
anonymous nature. 

• . When passing out surveys, suggest that students can take from 

the middle or bottom of the stack if they desire. 

• Do not have teachers pick up questionnaires from individual 
students. Have students place the surveys in a box before 
leaving the room. 

• If possible, consider obtaining the "passive consent" of parents. 
Active consent would require a written release allowing the 
student to take the survey. Passive consent requires the parent 
to be given a reasonable opportunity to refuse the child's 
participation. 



Logistics/ • The survey should not take longer than one class period to pass 

Administration out, complete and collect. 

• All students in the building who are to take the survey should 
take it simultaneously or student interactions could change the 
results. 

Inform the public of your intent lo administer the survey by 
letter, newsletter, or newspaper. 

Provide clear instructions to persons administering the survey. 
The person administering the survey must set an appropriate 
climate about the importance and confidentiality of the survey. 



Analysis • Answer sheets should be screened to exclude responses from 

students not taking the survey seriously, not following 
instructions, or faking impossibly high use. 

• Results should be compiled separately by grade for each 
question. A tally should also be made of the percent of 
students using any drug and the percent using any form of 
alcohol. 
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• Results for a grade should be compared with standards CVhat 
should be"), with other grade levels, with district totals, with 
national results, and with data from previous years. 



Follow-up 



Assess the sensitivity of reporting certain types of information 
in the community. 

Be prepared to defend the survey results. 

Use visual aids in presenting survey information. 

Release information to the community in a timely manner. 
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Adolescent AOD Indicator System 

Western Regional Center for Drug-Free Schools and Communities 
Northwest Regional Educational Laboratory 

Purpose 

The purpose of an adolescent AOD indicator system is to monitor the extent and nature 
of use over an extended period of time at minimal expense. The best indicators are those 
which are collected regularly using standardized definitions and which are more sensitive 
to changes in student behavior than to changes in how the data arc collected (e.g., 
reductions in police budgets may reduce the number of arrests independent of any 
changes in the number of juvenile offenses). 

This indicator system can contribute to your needs assessment and program evaluation, 
complementing student alcohol and drug use surveys and other outcome measures. 
Demonstrating outcomes using multiple measures strengthens the case that your program 
was effective. 

An indicator system can be implemented inexpensively. Most of the indicators described 
here are regularly collected for other purposes on a monthly or annual basis. Someone 
simply needs to extract the relevant records and prepare a summary worksheet. If 
records on previous years are available, include statistics for two yeaiS prior to the stan 
of your program. 

School Indicators 

Inappropriate school behaviors are an imponant sign that students are at risk. An 
effective Student Assistance Program should improve attendance and grades but reduce 
suspensions and other behavior incidents. Virtually all schools keep some records on 
student behavior. 

The challenge in interpreting trends in school indicators is to determine whether the 
changes were due to the effects of your program in reducing use or to changes in policies 
or enforcement. A crack-down in school policies and enforcement on drinking should 
initially increase suspensions and alternatives to suspension but as these policies stan to 
have an impact on student behavior, suspensions should decline. Of course, if 
enforcement was relaxed, that would also cause a decline in suspensions. 

Another concern is that the low incidence rate of some indicators make them unreliable 
or unstable, so differences observed from year to year may not reflect real trends. 
Extreme caution should be used with indicators with very low rates of incidence. 

Arrest Indicators 

Juvenile arrests reflect student behaviors outside of school. Police departments across 
the nation report arrests in a standard way under the Uniform Crime Reporting System. 
Monthly or semi-annual counts of various offenses by juveniles are easily obtained from 
your local sheriff. 
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One concern in interpreting arrest statistics is matching the area of jurisdiction and the 
service area of the school or school district. Another concern is the potential for other 
factors to effect reported arrests, including the funding level for law enforcement, 
organizational changes (e.g., adding a drug or juvenile task force or unit), and current 
arrest policies. 

Other Indicators 

There are a few other indicators that may be obtained locally, but are not generally 
available. 

Hospital emergency admissions. Adolescent hospital emergency admissions that 
involve substance abuse represent a potential indicator. Hospitals maintain case records 
on emergency admissions and may be asked to report to the state department of health 
and human services. Certified hospitals are often required to compute quality control 
indicators, but have considerable leeway in developing the specific indicators. In a 
metropolitan area, it may be possible to obtain indicators from a large health plan that 
serves a representative population. 

The problem with admissions records is that hospital officials admit that there is serious 
underreporting of drug or alcohol involvement. Doctors are reluctant to report drug or 
alcohol involvement and the symptoms of use are often obscured by injuries requiring 
immediate attention. 

Ambulance calls. Particularly in rural areas, ambulance calls may be more appropriate 
than hospital emergency admissions due to the large area the hospital serves. Ambulance 
services or paramedical organizations may be able to supply statistics on adolescent drug 
and alcohol related ambulance calls. 

Adolescent treatment. Some states require treatment providers to submit data on in- 
patient and out-patient services provided during the year. Summary statistics may be 
available from the state Health and Human Services Department, the state Office of 
Alcohol and Drug Programs, or the local service providers. 
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SECTION 4: PROGRAM PLANNING 
AND PROCESS EVALUATION 

An evaluation consultant was asked to help a school district design an evaluation approach for its 
prevention effort The district had an impressive array of programs and services. All the bases 
were covered: prevention, early intervention, referral, and aftercare. Several different activities 
were occurring in each of these areas. Nevertheless, the district had difficulty providing the 
cvaluator with a clear statement of the unifying goals and objectives these activities were 
intended to achieve, A closer look revealed a general lack of integration between the activities. 
There were discontinuities across grade levels, inconsistencies in the message presented by the 
various curricula being used, and inappropriate rivalries between groups involved in the various 
activities. The district had launched these activities piecemeal, over several years, and had not 
successfully confronted basic issues in a way which would have allowed it to build a coherent 
approach. District officials concluded that the opportunity to verify the achievement of 
meaningful results had largely been lost, as a resnU of deficiencies in planning. All energies had 
been devoted to implementing services, with evaluation considered after the fact. Evaluation is 
usefol when planning is carried out to ensure that services work toward a consistent purpose. 
Evaluation obliges the planner to view services as means to an end, rather than as an end in 
themselves. 

The systematic monitoring of program operations, as they relate to program plans, objectives, 
and goals, is called process or formative evaluation. Here, the focus is on the services and 
activities provided, who received them, when, how often, and so forth. Process evaluation 
studies the implementation and ongoing operations of the program. It allows the provider to 
describe how the intervention was delivered. This information is needed for several purposes, 
not least of which is to provide a background for the other major component of evaluation effort: 
the study of program outcome. 
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How to Plan a Process Evaluation 



ByJAOCFOLLARD 

Suppose you want to quantitatively document 
the openuion of your project or ptogranv but you 
doiftknowwiieretostart Youcangettheaitswers 
you xieed by designing and conducting a ^process 
evaluation.'' Here's how to proceed: 

The best way to begin planning a process 
evaluation is by thinking carefully about tfie stnic- 
ture or organization of the program that is to be 
evaluated 

Most programs, at a general level, have several 
siirdlar features (see Hgurel). Hrst, all programs 
have clients (for example, adolescents who are 
using alcohol or other drugs, or dysfunctional 
families) ard other resources (such as funding and 
office space) that enter into the program. Leaving 
the program are clients who have (we hope) 
benefited from die program's services, and idso 
program expenditures, such as personnel salaries. 

The program itself is represented as the central 
box in Hgure 1. Programs typically have a se- 
quence of assessment, treatment, and then re-as- 
sessment That is, when a client enters into the 
program, there is usually an assessment or screen- 
ing process where it is decided what program ser- 
vices ^ client should receive. Then the client is 
served by the pr o gr a m, and a variety of program 
services are deUvei^ed. Finally, at some point, there 
is a re-assessment of the clients condition to deter- 
mine whether more services are needed or if the 
client has been sufficiently helped. 

To make this structure more conaete, suppose 
that you want to coruiuct a process evaluation for a 
program that delivers counseling to ^milies of 
AOD-using adolescents. In the assessment phase, 
an examination is made of the family structure, its 
communication patterns, and so on. Then, several 
different services may be provided, such as work- 
ing to improve parenting skills, or instituting be- 
havioral contracting with the adolescent Finally, a 
re-assessment of the family functioning is con- 



ducted, looking at such variables as family 
cohesion or whetiier appropriate behavioral limits 
are beix^ set If s imporlint to note diat at the 
re-assessment phase, immediate changes in the 
lox^-term outcome of interest such as a reduction 
in AOD use, may not be observable In fact the 
final outcome nuy require some time before it can 
bemeasured, . / 

So what does^ of this have to do with planning 
a process evaluation? In Hgtue 2, several measures 
have been proposed that could be used in a process 
evaluation to describe the activities diaracteristic of 
each specific sti^ of the program's operatioiL For 
example, under the assessment category, two 
potential measures are the student's level of AOD 
use or the amours of coercion in the family's com* 
munication. These variables are ones that describe 
the incoxxting state of the program's clients. 
Measures of the number of parenting skills taught, 
or the spedfics of the behavioral contracts, mi^t be 
useful for describing the program's services. In 
general, it is important that all phases of the 
program's operations should be adequately 
measured. 

In effect Figure 2 is a blueprint for conducting 
the process evaluation. When you have created a 
diagram such as Figure 1, you have completed 
much of the planning for the process evaluation. 
Once you have developed the program's model, 
what remains for planning the process evaluation 
is to decide what measures best describe the ac- 
tivities characterizing various stages of the 
program's operation and how this information can 
best be gathered. While accurately gathering 
evaluation information is sometimes diffiailt (and 
you might consider getting some specialized help 
in that area from the Western Center evaluation 
stafO, you are now well on your way to developing 
a soimd and convincing process evaluation. 

For more information on process evaluation, 
contact Jack Pollard, 16004 N.E. Sixth Street, Bel- 
levue, WA 98008, (206) 562-7095. 
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Measures of 
Program Entry: 

No. of Families 

No. of Referrals 

Amount of Program 
Funding per Client 



Measures of 

Program 

Assessment: 

Level of Student's 
AODUse 

Amount of 
Coercion in Family 
Communication 



Measures of 

Program 

''Treatment"': 

Number of 
Parenting Skills 
Taught 

Specificity of 
Behavioral Contracts 



Measures of 
Intermediate 
Outcomes: 

The Enforcement of 
Appropiiate Limits 

Amotmt of 
Coercion in Family 
Commtmication 



Measures of 
Program Exit: 

Level of Student's 
AODUse 

Sibling's AODUse 

Treatment Cost per 
Familv 
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Evaluating Program 
Implementation 



By ROY GABRIEL 

Northwest Regional Educational Laboratory 

It is dear that alcohol and drug abuse has 
caught the nation's attention. Policy makers, 
legislators and parents are demanding action 
and results ftom school and community 
prevention efforts. While some of our schools 
have been fighting this problem through 
organized programs and activities for years, 
numy have net It's only through the recent 
Drug-Free Schools funding that many schools 
have been able to initiate efforts at helping 
reduce this problem. 

What does all this have to do with 
evaluation? One of the great evaluation minds 
of all time once said, "Don't try to evaluate 
something n it doesn't exist." Another said, "Be 
sure you know what it is you are evaluating." 
Both of these sages advise us to look carefully at 
the implementation of a program as well as its 
outcomes. 

There are usually three major reasons to do 
a process evaluation of the implementation of a 
program. One is to be sure that the program 
and all of its important components are being 
implemented according to plan. A second is to 
collect process information along the way to 
detect components needing improvement. 
Fmally, it helps identify particularly successful 
elements of a program which can be shared with 
others. 



Comprehensive programs in the fight 
against drug and alcohol abuse involve very 
complex sets of activities. The model we use in 
our work wiflx school teams is shown in Figure 
1. From Needs Assessment through Dissemina- 
Hon, these activities are connected and integral 
parts of the overall program. The snvooth 
implementation of each is important if positive 
student outcomes (i.e., a reduction in student 
use, or a dday in the onset of use) are to be 
realized* 

A helpful way to track the implementation 
of all program components is to have school 
tean\s periodically rate the progress of each 
program component on a scale such as: 

1. Nothing is happening yet in this area; 

2. We are just beginning to plan this 
component of our program (Initiation); 

3. Activities are starting to get imderway, 
but are in the early stages (Early Implementa- 
tion); 

4. All planned activities are in operation 
(Complete Implementation); 

5. Activities have been in place for some 
time with continuing support for the long term 
(Insti tutionalization). 
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Figure 1, A Comprehensive Model for Drug-Free Schools (adapted from Fox and Forbing (1987) 
Fighting Substance Abuse, in Our Schools) 
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One state's results are shown in Figure 2. 

How is this information helpful? In 
Prevention/ the largest percentage of schools in 
this state (nearly 40 percent^ have moved 
beyond plaxming and are into at least early 
implenventation of their activities. For 
Intervention, the largest percentage of these 
schools (nearly 40 percent) indicate that nothing 
is going on In this state. 

Does this discrepancy reflect the plans of 
local programs? Based on these ratings, should 
more resources and attention be given to 
Intervention activities? Or should more 
encouragement and support be provided to 
increase the implementation of Prevention 
activities to 100 percent? 

Figure 2 includes only two components. A 
more complete picture would be gained by 
looking at the le\^ of implementation of all key 
activities in the program: Needs Ass^ment, 
Policy Development, Parent Involvement, 
Evaluation, Dissemination, etc. Are they all 
moving along according to plan? Is the picture 



different for elementary, junior high and high 
schools? Is it changing over time? Asking staff 
involved in the program to do these ratings 
every few nx>nths not only can answer these 
questions, but can inaease their awareness of all 
tixat is going on in the program. 

Information such as this also helps in the 
luvderstanding of outcome data. If anticipated 
improvements in students' behavior (e.g., 
attendance, disdplinaiy referrals) or reduction 
in their alcohol and drug use do not occur, 
perhaps some activities that were planned didn't 
occur; or if fl\ey did, they didn't occur the way 
they were intended. 

Remember the sage words of the second 
famous evaluator quoted above: "Know what it 
is you are e^^uating.•• Systematically 
describing the implementation of all program 
components can help. 
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Figure 2. Level of School Implementation 
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Sample Process Indicators in Evaluating 
School Programs 

Once the purpose of and important audiences for the evaluation are determined, the school team 
must think about the information needed to fuel program improvement efforts and demonstrate 
impact Once these information needs are identified, various methods for obtaining the data can 
be considered. Here arc some examples of general indicators typically found useful in the 
evaluation of school programs in alcohol and other drug education. 

• School enrollment 

• Number of student participants in the program 
/ ^ • Background characteristics of students 

• Activities in the program 

• Number of activities per week 

• Hoxxrs per week of planned activities 

• Students' participation in other school programs 

• Number of school staff involved in the progi^am 

• Community members participating in the program 

• Resources allocated to the program 

• Distribution of resources throughout the program 
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California SAP Study Yields 
Surprising Findings 



By JACK POLLARD 

Southzvest Regional Laboratory 

Preliminary findings are now being 
reported from an evaluation of student 
assistance programs (SAPs) currently 
being conducted in California. Many 
of the evaliiation findings have 
surprised both the directors of the SAP 
projects and the researchers conduct- 
ing the evaluation. 

- The evaluation is funded by the 
California Department of Alcohol and 
Drug Programs (DADP). It examines 
the functioning of five model 
demonstration projects located in eight 
school districts scattered throughout 
California. The SAPs were funded by 
DADP in 1989. ,The evaluation effort is 
xnanaged by the Southwest Regional 
Laboratory (SWRL). 

One of the most important prelimi- 
nary findings is that about 50 percent 
of all the students entering the SAPs 
are self-referrals. That is, the students 
have decided on their own initiative to 
utilize the services of the SAPs at their 
schools. This high proportion of self- 
. referrals seems to indicate that the stu- 
dents have strong positive attitudes 
toward the SAPs. 

Another important finding was 
that 35 percent of all of the students 
referred to the SAPs reported the exist- 
ence of a serious personal problem. 
These problems often focused on an 
immediate personal crisis, such as 
parental alcohol/drug use, an im- 
pending family divorce or breakup, 
sexual abuse, physical abuse, severe 



depression, delinquency, personal 
trauma (e.g., rape), or gang activity. 

Even more interesting was the fact 
that the more severe the student's 
presenting problem, the more likely 
the student was to have referred him- 
self or herself into the SAP. In other 
words, when students found them- 
selves facing a serious personal crisis, 
they were quick to take advantage of 
the SAP services. Again, this pattern 
of results indicates that the students 
are placing a great deal of trust in the 
functioning of the SAPs. 

The SAP programs appeared to be 
doing a good job in effectively and 
quickly serving their students. Most 
students received SAP services within 
two days of their iiiitial referral, and 
the students received an average of 
nearly seven i;ervice events." A "ser- 
vice event" can include individual or 
group counseling services, interven- 
tion with the family, or referral to out- 
side social service providers. When 
the student reported a serious prob- 
lem, the time for the initial SAP service 
was less than one day, and the average 
amount of services nearly doubled. 

Some differences were found in 
how the SAPs functioned at the middle 
and high school levels. For example, 
while at the high school level there was 
a greater proportion of girls participat- 
ing in the SAPs, at the middle school 



level the numbers of boys and girls 
participating were about equal. 

Also, the typical presenting prob- 
lem and the typical referral source 
were somewhat different at the two 
school levels. At the high school level 
the students primarily referred them- 
selves into the SAP. At the middle 
school level, other referral sources, 
such as teachers and classified person- 
nel, were just as important. Also, mid- 
dle school students were more likely to 
be assessed as having behavioral and 
family problems, while the high school 
students were much more likely to be 
assessed as having problems in their 
social /personal relationships or prob- 
lems with alcohol or other drug use. 

The SAP evaluation is continuing 
through July. This year the evaluation 
focuses on measuring the impact of the 
SAPs' operations on the students' per- 
sonal, school, and social functioning. 
Changes in functioning of the students 
participating in the SAPs v^U be com- 
pared to other students at school sites 
where SAP services are not available. 
Findings from this phase of the evalua- 
tion should be available in August. 

The report documenting these find- 
ings is Student As'iistance Program 
Demonstration Project: Process Evalua- 
tion. If you are interested in receiving 
a copy of the report, contact Jack Pol- 
lard, Southwest Regional Laboratory, 
4665 Lampson Ave., Los Alamitos, 
California 90720, (310) 598-7661. 



'Half of all students entering 
the SAPs are self-referrals/ 



-Jack PolUrd 
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'Ask Dr. Evaluation' 



Scope, Purpose, Design 
Affect Evaluation Expense 



By FLORA YEN 

Northwest Regiond Education 

/Q. OuriistricthaputabtofcffoHandmo^ 
umdadassbtanccprogmmforabnast thruym nnauK 
East spring the 9chool board asked us ta 
shouringv^iiffererux the program has tna^ 
studeitis. Canyougpe%isanestsmaUofwhatitxDOuid 
cxjst to evaluaU our SAP and what items these costs 
nd^t cover? 

A. The cost depends a great deal on the scope 
and size of the progranv the purpose for doing the 
evaluation, the complexity of the evaluation 
design, and your access to evaluation expertise. 
Because these things vary so mucii from one district 
to another, it would be hard for me to give you a 
figure without knowing more about yoor program. 
I have, however, used two approaches in estimat- 
ing evaluation expenses* The first approach is to 
get a ballpark figinie based upon a percentage o f the 
total program budget and from there find out what 
range of evaluation activities this amount would 
siq>port Qearly, the amount of money available 
will determine the flexibility and sophistication of 
the evaluation desigru The second approach is to 
identify the purpose of the evaluation and the tasks 
that will be needed ^x> achieve that ptarpose; es- 
timate staff and consultants' time and other expen- 
ses to do these tasks; and then add up the individual 
items. These approaches need not be mutually ex- 
clusive but can be combined to arrive at a 
reasonable cost estimate. 

A Ballpark Figxire 

There is no fixed percentage that all evaluiators 
use to determine costs. The percentage in many 
cases depends on the size of the total budget of the 
program. A program with a budget of $500^, for 
example, will most likely require more sophistica- 
tion in evaluation strategies than, say, a program 
with a budget of $5,000. Consequently, the percent- 
age will vary. 



In the Evaluation Management Handbook, M. 
Raupp and FA. Kolb of NETWORK, Inc., advocate 
that the budget for evaluation activities be some- 
where between6 percent and 10 percent of tiie total 
programbudget For example, if the funding level 
of yotar progiam is $50,000, estimate about $3,000 
to $5,000 for evaduatioiu From my experience in 
evaluating educational programs, this 6 percent to 
10 percent range is realistic and much needed 
Quite frequendy, program cnanagement underes- 
timates the anru)tmt of txa\e and effort needed to 
undertake aiui complete evaluation tasks. 

For a pilot or demonstration program, however, 
the evaluation costs can be higher. Since the proce- 
dures, strategies, and n:utenals being used are new 
or the target group involved is x\ew, the biirden of 
proof is upon the program to show whether such 
irmovations actually work and whether they 
should be supported in the future. In their March 
1990 anxu)xmcement for applications uiuier the 
Community Partnership Demor^stiationGrantpro- 
gram, the Office for Substance Abuse Prevention 
(OSAP) suggested that applicants target 10 percent 
to 15 percent of tlieir proposed program budget for 
evaluation activities, although explaiutions are 
needed to justify the amoimt requested. 

Since the SAP concept has been around for a 
couple of years now, with some reports showing 
that it has been tiseful in a number of school dis- 
tricts aroxmd the country (thotigh much still needs 
to be done to document SAP outcomes and assess 
its long-term impact), I would go with the 6 percent 
to 10 percent range of your program budget as a 
rough estimate for covering the expense of evalua- 
tion. 

At this point, let me describe the sequence of 
activities that ustially occurs when evaluating an 
'Educational or social program. These are: 

1. Deciding on the questions that program staff 
and management (as well as other stakeholders) 
want to have answered about the program (in your 
case, what difference the SAP made for students m 
your school) 
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2. Working out an evaluation plan (including 
design, instruments, timelines, and assignment of 
tasks) that would beappropriate and teaisonable for 

the questions being asked 

3* Collecting information to address the ques^ 
tionsposed 

4* Tabulating and analyzing the information 
collected 

5. Reportir^Ae findings 

To ittustrate this approach further, let me give 
you an kleaofwhatltiunkyou might get for $5/XX} 
if you were to hire an outside evaiuator. The 
cvahiator win w*d to darifyTaskl with you, your 
sta£f,axid theboard, and ^^d the greatest portion 
of her or his time doing Tasks, 2, 4, and 5. It is 
usually recommended ^t an evaiuator make at 
least one onsite visit to the program being 
evaluated. Seeing a prognun in action gives an 
evaiuator a better understanding and context 
within which to plan evaluation activities and to 
analyze and intetpret the information collected 
Ibr a $5^ contract, it is most likely that you and 
your staff will do the major portion of Task 3, 
although dus will depend on the outside evaiuator. 

In any event, the $5^ is a ballpark figure to 
use in estimating costs. With tius figure, you can 
contact several professional evaluators and ask 
than to giveyouaoostbreakdownof what they can 
provide. If you are fortunate enough to have access 
to free technical expertise or have in-house 
evaluators, certainly check with these sources first 
and get a cost estimate from them. 

The Evaluation Budget 

A budget for program evaluation usually in- 
cludes these items: 

• Salaries of personnel (staff and consultants) 

• Travel expenses 

• Supplies 

• Duplication 

• Telephone 

• Postage 

• Computer services 

• Facilities cost (if applicable) 

• Indirect costs (if applicable) 



Using this second approach in estimating 
evaluation costs, you would need to know staff 
salaries and how much staff time will be needed. In 
addition, you will need to find out ahead of time 
what the approximate daily rate for an outside 
evaiuator or consultant is, ^iild one be needed. 
The rate may vary depending on the organization 
the evaiuator works for, the individual's education- 
al background and experiences, and the region 
where she or he is based. A daily rate of $150 to 
$250intheWestisnotuncommon. Again,itwould 
be a good idea to contact several potential 
evaluators and get an average daily rate to include 
in your budget Expenses for interviewers or 
coders and other support staff for the evalutator also 
should be figured in. 

Travel expenses canindude mileage reimburse- 
ments, plane fare if applicable, lodging, and per 
dient The more time required of an evaluatar on- 
site, the hitter the travel expenses. Persomieland 
travel costs tend to repr e se nt ihe major ponions of 
anevaluationbudget The other costs are generally 
minor in comparison. 

I would suggest that you check with your dis- 
trict and with your state educational offices, as well 
as nearby iitttitutions of higher education, to deter- 
mine what kinds of technical assistance in evalua- 
tion you have access to and any costs that might be 
involved. Also, all school districts in the Western 
states can request some technical assistaiice in pro- 
gram evaluation at no cost to them from the 
Western Regional Center. School districts in the 
Pacific Northwest, Alaska, Hawaii, and the Pacific 
should call the Portland office, (800) 547-6339, ext. 
497. School districts in Northern California and 
Northern Nevada should call the San Francisco 
office at (415) 565-3000, and the districts in Southern 
California and SouthemNevada shotild call the Los 
Alamitos office at (213) 59&-7661. Although Center 
staff cannot evaluate your program for you, they 
can offer suggestions and resources and answer 
some questions over the telephone. Depending on 
your location and their schedules, they may be 
available in some cases for onsite consultations. 
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SECTION 5: OUTCOME EVALUATION 

Questions about outcome (sometimes referred to as summative evaluation) are likely to be the 
primary reason for initiating evaluation activity. "What is your success rate?" is a question often 
posed to program officials. As the preceding discussions demonstrate, there are other important 
dimensions of evaluation activity as well, but outcome is likely to remain the chief interest of 
policy decision makers and the general public. 

The following is a list of possible "outcome" indicators in a school-based AOD prevention 
program: 

Knowledge about the varieties and effects of alcohol and other drugs; 
Attitudes toward alcohol and other drug use; 
Students' use of alcohol and other drugs; 

Feedback from students, teachers, and parents on program effects; 
Students' attitudes toward school and learning; 
Student achievement; 
Student attendance; 

Disciplinary referrals related to alcohol or other drug use. 

Methods for collecting this information are similar to those discussed in earlier sections. Student 
use surveys, for example, may be used for both needs assessments and outcome evaluation. 
Included in this section are further discussions of student use surveys used to measure program 
outcome, an example of a report describing outcomes of prevention efforts in the state of 
Washington, the need to document the effects of Student Assistance Programs, and a case study 
approach to documenting outcomes. 
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'Ask Dr. Evaluation' 



Student Surveys Can Help 
Gauge Program Success 



EDITOR'S NOTE: This issue, we begin a new fea- 
ture, "Ask Dr. Evaluation/' In this regular colimn, 
members of the Western Centet^s Evaluation TaskForce 
xaitt respond to evaluation questions most often asked of 
Western Center sti^ in the fidd. The column begins 
loitk a response from Roy Cabrid, associate director for 
evaluation, to a question he and his co^oorkers hear 
frequentfy from frontAine educators and A&D profes- 
sionals. If you have other evaluation questions you 
would Uke addressed, please serui them to Roy Gabriel, 
Western Center for Drug-Free Sck ^s and Com- 
munities, 101 S.W. Main Street, Suite JOO, Portland, 
Oregon 97204. 

Question: We've invested heavily in a preven- 
tion curriculum in our district The publisher came 
out and trained all of our teachers two years ago 
and we were very enthusiastic How do we know 
if s working and that all of this effort we are putting 
into prevention is worthwhile? 

Answer: A question like 'IHow do we know it's 
working?" begs another, more fundamental one: 
'TVhat is it supposed to accomplish?" Reputable 
prevention curricula are usually built around 
specific goals and objectives, such as: 

• Increase awareness of the harmfulness of al- 
cohol and drug use 

• Identify healthy alternatives for having fun 

• Develop greater awareness of the importance of 
careful decision izuking 

Hopefully, you were aware of and endorsed the 
objectives whm you pvirchased yoxir curriculimi. 
In fact you might get from the publishers any 
research or evaluation evidence they have that their 
curriculum "works/' When such evidence exists, it 
is a good idea to include, but not be limited to, 
similar methods and instruments used in those 
evaluation studies. Even if their findings from ear- 
lier field tests or pilot sites are encouraging, how- 
ever, it is still best to conduct your o wn evaluation 
to see if the curriculum works with your students in 
your school and commimity. 

Checking the ways in which your teachers actual- 
ly use the materials is also recommended. No matter 
how good the track record of the curriculum mav be. 



or how appealing the materials look, if the teachers 
do not inclement the curriculum as designed, itcan't 
be expected to achieve the desired effects* Ask your 
teachers questions like: 

• How well dots the prevention curriculimi fit in 
with other subject area curricula? 

• How many hours per week do you use the 
materials? 

• Which grade levels are involved? 

Chances are good that the curriculum includes 
brief quizzes or tests to detemuiie whether the stu- 
dents have mastered the content and attained the 
objectives. By administering these tests before and 
after the curriculum has been implemented, you 
can demonstrate whether the objectives have been 
met In a senscr this provides some evideruire that 
the curriculum ''is woridng." 

But this is only a small part of whether or not 
your overall prevention eftbrt is working. The 
school board, the local media, parents, community 
members, and others will want to know if alcohol 
and drug use is declining. A carefully designed 
survey of student AOD use is the most common 
mechanism for that determination (for more details 
on surveys of student AOD use, see the Western 
Center publication. Surveys of Student Alcohol and 
Other Drug Use A Consumer's Guide, available free 
from the Resource Center). Repeating the survey 
each year can provide a wealth of information on 
the trends in use of alcohol and other drugs in the 
school and commimity. 

But even if the results of such a survey are 
encouraging, how do you know whether it was the 
prevention curriculimi that caused the change? 

Well, chances are you won't Prevention is a 
massive effort Yes, tiie classroom curriculum is 
part of the effort but so are other activities spon- 
sored by students, parents, community coalitions, 
and so on. 

In short, without rigorous experimental design 
conditions — so rarely feasible in the real world of 
public and private schools — we cannot pinpoint a 
single cause of a well-documented effect The good 
news of AOD use declining is more likely an ac- 
cumulation of effects of a well-articulated, com- 
prehensive prevention strategy. 



The question posed here was initially one deal- 
ing with the e^ectiveness of the curriculum. 
Specific quizzes or unit tests, tied directly to .those 
objectives, are tlw best xneasiues of that effective- 
ness. If yDurovendlpreventionoblectivesaremore 
^bal, don^ t look to the curriculum to do more than 
it is designed to do« 
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Washington State Makes 
Progress in Prevention 



By ROY GABRIEL 

Northwest Eegional Educational Laboratory 

Areocnt survey of n«riyl8^studentsacross 
the ftite of WasWi^ton thows positive progress in 
that state's eftorts to redticc the incidence and 
prevalence of alcohol and other drug (AOD) use 
among its sdiool-aged youth. H^e survey also in- 
dicates that much woric stiU needs to be done to 
attain the goab of drug-free schoob and com- 
munities thnni^iout the state State Superinten- 
dent of Public Instruction Juditfi Billings calls for 
school and communis personnel to ^continue their 
collaborative efforts to fi^t the war on drugs-a 
battle that is far from won.'' 

The survey points to ''good news and bad news" 
about students' attitudes and use patterns, accord- 
ing to Billings- The •'good news" is that use rates 
aregeneraUydown, related -'riskfactonT appear to 
beimpioving, and the influenceof school programs 
is increasing. The 'T>ad news" includes some in- 
creases in heavy use rateS/ particularly at the 
eighth-grade level. 



Among the stud/s key findings is that, com- 
pared to 1988 statistics, fewer Wastogton students 
report having ever tried alcohol or drugs. The 
decline is most dramatic in grade six Whencotiv- 
pared with their grade-level peers in 1988, sig- 
nificantly fewer students have yet to experiment 
with alcohol or other drugs. As shown in Figure 1 
below, only 33 percent of sixth-grade students in 
1990 had ever tried alcohol, compared to more than 
51 percent of sbctti-greders in 1988. As Billings 
notes: 'Tltis delayed or«et will pay huge dividends 
for Washington if current prevention efforts are 
sustained." 

Moderate to high levels of AOD use have also 
declined since 1988, but not as significantly as initial 
experimentatioiv as dted above. In fact, at eighth 
grade, there is evidence of higher use rates, par- 
ticularly of alcohol 

Despite the delay in onset of use, alcohol abuse 
continues to be a problem among Washington's 
students- Although slight dediixes in heavy use 
occurred at grades sbc and 10, a higher proportion 
of eighth-graders dnmk heavily in 1990 than in 
1988. 




1988 
1990 



Gradd 

Figure 1. Percent of Washington students having 
ever tried alcohol in 1988 and 1990 by grade. 
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Figxire 2. Percent of lOth-grade students learning the 
most about the dangers of alcohol or drugs from 
various soiutres* 



In 1988, 55 percent of Washington's eighth- 
grade students reported heavy use of alcohol, con- 
suxningeidier moderate amounts daily or excessive 
quantities at least monthly C'binge drinking"). In 
1990, this percentage is nearly 7 percent Across 
grades six tiuougji 12, the survey projects that ap- 
proximatdy 45,M0 students are heavy drinkers. 
The particularly high rate in grade eight has 
prompted state administrators to call for greater 
emphasis on prevention and intervention strategies 
in the state's middle schools. 



In addition to student use rates, the 
survey also probed related behaviors 
known to be associated with alcohol and 
other druguse. Inparticular,manyof the 
"^risk factors^ sjnithesized by David 
Hawkins and others are represented in 
the survey* Some of the major findings 
were: 

• Students' opportunities to use alcohol 
and other drugs (friends who use, 
availability at parties, etc) are at or above 
those in 1988. 

• Students perceive their parents to be 
subtly more disapproving c f alcohol or 
other drug use than in 1988* 

• Student see great risk in drug use, but 
far less in heavy drinking* This percep- 
tion of riskinoeases with grade level, 
high school seniors see the greatest risk 
inAODuse* 

• In general, students who are active in 
extracurricular activities are sli^tly less 
likely to use alcohol or other drugs* For 
some types of activities, such as sports, 
however, this pattern is not so dear* 

• Students who aspire to go to college 
use alcohol and otho- drugs far less thw 
those who do not* 

The survey also yielded important 
feedback for the influence of school 
prevention programs on student use of 
alcohol and other drugs. As in 1988, and 
in even higher proportions in 1990, 
Washington students find school the 
most important source of ix\fonnation 
about alcohol and other drugs. The in- 
crease in school influence appears to be 
at the expense of the popular media, 
which declined from that of 1988 at all 
grades. 

At all grades, half or more of the stu- 
dents indicated that school was their 
major source of information on alcohol 
and other drugs. 'Tamily" was dted 
next, by 20 percent to 30 percent of the 
students at each grade* The influence of the media 
was tiurd, reported by 10 percent to 20 percent of 
the students at each grad^?. While media influence 
increased with grade level, it was significantiy less 
than that reported in 1988. Rgure 2 displays the 
results for Washington's lOth-grade students in 
1988 and 1990. 

Students who reported they learned a lot from 
their drug education program were much less like- 
ly to report moderate to high alcohol or drug use. 
Suggestive evidence of the effectiveness of school 
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programs across the state was obtained through 
correiatiorudartalysisof the survey information. 

As shownin RguxeS/ there is a ooitsistent trend 
showing that students who learned a lot ftom their 
drugcducationdasses report lower AODuserates- 
As in all corrdational studies, however, this does 
not prove that greater infomuition gained throu^ 
sdiool programs is directly responsiUe for bwer 
AODuseratcs* In fiact, the causal link could go the 
other way^-^udents who tend to use alcohol or 
other drugs less miy be more likdy to team more 
indrugeducationclasses. In dtherevent, however, 
the liitk between AOD use rates and information 



gained through school programs is strong and in 
the logical direction. 

The survey was coxulucted by the Northwest 
Regional Educational Laboratory in both 1988 and 
1990. Hie current report is based on a scientific 
sample of 176 schools, stratified by geographic 
region axui size/rurality within each grade level 
For further information or a copy of the full report, 
contact Debbie Koss*Wamer, [7rug*Free Schools 
and Communities Program Supervisor, 
Washington Office of the Superintendent of Public 
Instruction. 
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Figure 3. Percent of students at each grade reporting moderate to 
high alcohol use by the amount learned from drug education classes. 
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How Effective Is Your Student 
Assistance Program? 



By DENNIS DECK 

Northwest Rtgioml Educational Laboratory 

During the 1980s, Student 
Assistance Programs (SAP) have 
become a popular iimbrella for 
intervention and prevention activities 
among junior and senior high schools. 
There are now two nd.tional 
oiganizations and an educational 
jotumal devoted to such programs.^ 

Despite this popularity, there are 
some signs that local, state, and national 
policy makers are not ready to embrace 
SAPs without some evidence of 
effectiveness. A literature seajxh turned 
up virtually no relevant studies, yet SAP 
coordinators feel that their programs are 
effective. One of the few reports from 
an SAP to include any impact data 
comes 



from Westchester County, NY^. Figure 
1 compares self-reported drug use 
before and after students participated in 
a school support group. The results ^re 
striking, the percent of participants 
using alcohol in the last 30 days 
dropped nearly in half and the percent 
using marijuana dropped even more. 

Unfortunately, (he report does not 
describe how the study was conducted 
so we do not know how carefully the 
evaluation was implemented or how 
one might replicate it. However, if they 
are available for your program, this kind 
of behavioral change data is very 
convincing to a school board. 




Fig\ire 1. Decline in 30 day use of alcohol and marijuana among 
support group participants (Westchester County, 1984). 
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' An Evaluation Framework 

The first step in preparing for an 
evaluation is to be clear about the 
objectives of the program. Table 1 lists 
examples of the typical prevention and 
intervention objectives expressed by 
SAP coordinators. These exarrples 
emphasize students, but also include 
school staff, parents, and others. 

It is helpful to distinguish between 
process objectives /mvolved in the 
implementation of the program and 
outcome objectives specifying the 
expected impact on student b^uivior, 
attitudes, or knowledge resisting from 
program activities* Most evaluations 
use both of these types of data. 

Process Evaluation 

A good process evaluation identifies 
the degree of implementation of 
program components and the level of 
service provided. Most important 
during the early stages of the program, a 
modest investment in process 
evaluation can yield great benefits in 
improving your program. 



Impact Evaluation 

An impact evaluation measures 
changes between a pretest and posttest. 
To attribute these changes to the 
program, it is important to show that 
similar changes were not observed for a 
comparison group that did not receive 
services. The relevant outcome measure 
for a component of the program might 
be an alcohol and drug \ise survey 
(behavior), a self-esteem survey 
(attitudes), or a test on the effects of 
drugs (knowledge/skills). The program 
should be fully implemented and 
working smoothly before an impact 
evaluation can be expected to show 
positive results. 

Practical Guidelines 

There is increasing pressure at the 
national level to require more 
evaluation of programs funded through 
the Drug-Free Schools and 
Communities Act, but so far no 
guidelines for such evaluations have 
filtered down to the state and local level. 
In lieu of federal guidelines, here are 
some general suggestions to guide 
evaluation planning. 



Prevention 



Intervention 



- Process - 

Present strong drug-free message for all school activities Train teachers in identification and referral process 

Develop a strong school policy Establish support groups In all Jr. /Sr. High schools 

Select a good prevention cuiriculum for all grade levels Provide appropriate referrals for all students 

Train teachers in use of curriculum suspected of drug or alcohol involvement 

— Behavioral Outcomes — 

Prevent use of lUidt drugs Reduce use of alcohol and other drugs 

Delay onset of use of alcohol and other drugs Increase the number of students obtaining 

Increase student attendance at alternative activities assessments and treatment 

Reduce drug-related sxispensions and expulsions Reduce school vandalism and other behavior 

incidents 

Reduce juvenile DW! and other drug-related arrests 

- Attitudinal Outcomes - 

Improve student feelings of self-worth and ability to Improve seU«worlh of participants 

control own destiny Improve school attitudes of participants 

Improve st\ident attitudes toward school and school work 

- Knowledge /Skills Outcomes - 
Increase student knowledge about the effects Impi ove CPA of students served by the program 

and dangers of alcohol and other drugs Reduce unexcused absences by students served by 

Improve student dedsion-making skills the program 

Improve student refusal skills Reduce the schcwl dropout rate 

Increase parent and staff awareness of the local problem Increase parent and staff awareness of possible 

interventions 
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Make a long term commitment to 
evaluation. Establish a long tenn 
planxung process that considers needs 
assessment and evaluation results in 
program decisions. Invest a portion of 
your budget (some funding agencies 
require 5-10%) and other resources on 
evaluation. 

Develop as strong an evaluation 
plan as resources allow. Prepare a 
statement of program goals, objectives, 
and standards to serve as the basis for 
the evaluation plan. Develop the plan 
as early in program development as 
possible. Seek professional evaluation 
assistance from the district evaluation 
department, university evaluation 
departments, or regional training 
centers. 

Implement a process evaluation to 
help improve the program and 
document the level of service. 
Maintain a recordkeeping system that 
tracks the progress of students through 
the program and that yields summary 
infomnation about what services have 
been provided. It is important that the 
system identify students who have fully 
participated in components of the 
program, not just the total referrals. For 
example, if 100 students attended a 
support group, that shows more 
effective implementation than if 100 
were only referred to the program and 
attendance at the group was unknown. 

Administer a staff survey, focus 
groups, or on-site observations to 
determine how well the program has 
been implemented and whether school 
team activities are consistent with the 
program goals. 

Implement an impact evaluation* 
Administer short-term outcome 
measures for key program components 
(e.g., student follow-up survey for 
support group participants). Focus on a 
different component each year. These 
evaluation studies should start after the 
component has been fully implemented 
and should include pretest and posttest 
measures for both the treated group and 
a suitable comparison group. Monitor 
behavioral indicators to determine long- 
term effects and start data collection 
before the program has taken effect. 



Adnunister a student-use survey on 
a two- or three^ear cycle. Admin- 
istration and sampling procedures 
should be standardized. Keep annual 
records of the total drug-related school 
suspensions and juvenile arrests. Also 
record the current enforcement 
environment, since both suspensions 
and arrests are sensitive to changes in 
policies, funding, and enforcement 
crack-downs. 

Summarize and disseminate the 
results to appropriate groups. Those 
responsible for nuking funding and 
support decisions (e.g., school boards, 
district administration, business or 
community committees, and funding 
agents) will be especially interested in 
your results. Those involved with the 
program (e.g., school staff, service 
providers, students, and parents) should 
receive feedback on how well the 
program is being implerr ?»nted. 

Other educators considering 
initiating SAFs are interested in effective 
programs and nuy be reached through 
regional and national journals and 
conferences.- 

Thc Time is Now. Your SAF n\ay 
currently enjoy widespread support, but 
that support will not continue 
indefinitely without evidence that the 
program is providing a high level of 
services and can demonstrate outcomes. 
Now is the time to develop an 
evaluation plan for the next three years. 
When the questions about effectiveness 
and program continuation begin to 
surface, you will be prepared. 



^National Association of Leaderslup for 
Student Assistance Programs (NALSAP) 
P.O. Box 21838 
Milwaukee, WI 53221 

National Organization of Student 
Assistance Programs and Professionals 
(NOSAPP) 

250 Arapahoe, Suite 301 
Boulder, CO 80302 
(800) 972-4636 

Student Assistance Journal 
Performance Resource Press 
2145 Crooks Road, Suite 103 
Troy, MI 48084 
(313) 643-9580 

^An implementation manual for this 
model is available in: Preventing Alcohol 
Problems Through a Student Assistance 
Program: A Manual for Implementation 
Based on the V/est Chester County, New 
York, Model, National Institute on 
Alcohol Abuse and Alcoholism, 1984 
(DHHS Publication No. ADM 84-1344). 



From Western Center Nru)s, Volume 2, Issue 2, March 1989 

5-10 



Section 5 



Evaluation Resource Notebook 



Case Study Outline 
Student Assistance Program 

Purpose: The purpose of a case study is to provide to the reader an understanding of the 
operations and outcomes of a student assistance program in a particular school or district. 
The outline could be modified for any other program developed to meet a specific goal. 
The emphasis is on description rather than comparison. Comparisons with unserved 
youngsters is assumed to present many problems-e.g., finding a school without a 
program. A clear description of the school situation, why the program was introduced, 
the program and its history to date, and program participants is useful to others as they 
begin their programs. We can learn outcomes of those who enter the program from the 
case study. In a case study report, we do not know what the outcome would have been 
had there been no program. 

Background: The information in the background provides the reader with an 
understanding of the context of the school and community. It is assumed that within 
similar contexts, the program efforts may be generalizable. Information that could be 
included: 

Enrollment of the school; percent minority; income and occupational level of 
families; urbanicity; school administration tenure; teacher characteristics; school 
climate. 

Development of the Program: This places the program in the context of the school. 
Description of the need for the program, program goals and objectives; persons involved 
and so forth will help the reader understand those conditions that led to the development 
of the program. Descriptions should include: 

The forces that prompted the program (e.g.» data, student initiative, parent 
initiative, accident, teachers went to training); the program model; changes to the 
model to fit school needs; persons involved and their roles; time and financial 
support required; training provided and who; involvement of the community. 

Description of the Student Population Served by the Program: Here the case study 
begins to merge the descriptive data with descriptions of the program. These descriptions 
will allow the reader to draw inferences made on data versus narrative. Generally the 
best descriptions are those that follow the students through the various options available 
in the program. Examples of data descriptions that are helpful are presented below. You 
will adjust your presentations to match your own program design, 

Informal Program Assessment: In addition to the information presented on student 
progress in the program, you will want to present in your case study any information 
available from faculty, students, staff, or parents about the program. This information is 
best obtained in a brief survey but may also be obtained through informal interviews with 
representatives of these groups. 
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Student Case Studies: A report of one or two students adds highlights and personality 
to the report. Any individual description should be disguised sufficiently so that other 
students or faculty may not easily identify the student. The following is an example: 

Christa A* 

Christa is a 15-year-old female in the sophomore class in our high school. She 
came to the attention of the Student Assistance Team (SAT) through a teacher 
referral. Christa*s fifth period teacher noted on the behavior checklist a concern 
that Christa was continually coming late to fifth period class. She also noted a 
decrease in her performances on weekly quizzes. 

Christa's teacher also noted that she had talked to her about the problem, getting 
little explanation for the behavior. The teacher was able to get Christa to sign an 
agreement to attend every class on time for the next two weeks and work with the 
teacher to bring her work up to "B"-her usual grade. 

When Christa continued to be late and absent two days of the next two weeks 
without excuse, her teacher sent the referral to the SAT. 

The SAT team decided to send out behavioral checklists to Christa's other 
teachers. They also reviewed the official attendance and grade report for Christa 
over the last two years. 

The following was noted: 

Christa was a transfer student from -x smali iiigh school of 300 students, located in 
a rural town. Previous grades were all A's and B's with a GPA of 3.35. Christa's 
record indicated she had been active in Junior Rally and the track team. Notes 
from the transfer interview indicated that her mother had moved due to her 
divorce. There are two other children, giris ages 6 and 1 1 in the home. 

The SAT team reviewed the behavior checklists. Christa's first and seventh 
period teachers noted attendance and tardy problems. They also noted a decrease 
in the quality of school work. The SAT agreed to call the mother to express 
concerns and see if she had noticed that Christa seemed very moody and lacked 
patience with her sisters. Christa's mother attributed that to typical teenage 
behavior. 

One of the SAT members, who had Christa in her class the previous term, was 
asked to visit with Christa and her counselor. Christa admitted she had tried pot a 
couple of times but denied that it was any problem. Christa agreed to a new 
behavior contract and agreed to attend an awareness class in an outside 
community agency. 

Christa missed the next two days of school. When called at home, her mother 
indicated that Christa had come home intoxicated. Christa's mother was unaware 
that she had missed any school. Christa's mother agreed to meet with Christa and 
the SAT. The meeting involved the SAT members-the SAT leader (a teacher), 
Christa's counselor, the vice principal, another teacher, Christa and her mother. 
The SAT reviewed her attendance, behavior in class, her own admission of drug 
use, and her failure to follow her own contract. Christa and her mother agreed to 
sign the releases of information, Christa went immediately to an outside agency 
assessment program. 
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The assessment indicated that an outpatient treatment program was the preferred 
method of dealing with her early stage of abuse/dependency. Christa agreed to 
attend a support group within the school for three days a week. This support 
group emphasized staying away from alcohol and drugs and developing 
alternatives for having fun. 

Christa attended the support group for one semester. A check of grades, 
attendance, and behavioral reports at six months and then one year later indicated 
satisfactory grades, no behavioral reports and five absences within the year. 



John B. 

John B. is a 16-year-old male in high school. He came to the attention of the SAT 
through contact with an outside treatment agency, with permission and 
cooperation from his parents. John was involved in an assessment by the 
treatment agency after an incident that involved law enforcement John was 
picked up by the police with other non-district students after smashing windows 
of cars in a school parking lot. John was under the influence of substances at the 
time. The juvenile worker assigned to his case recommended a professional 
assessment to the parents. Upon completion of the assessment, it was 
recommended that John be admitted for treatment. At that point, the SAT team 
became involved. 

The SAT team became involved with John's case immediately for three reasons. 
While John was in treatment, there would be time and structure for him to keep 
up with 1-2 classes, and the possibility of receiving credit for other work done in 
treatment. In order for John to keep up and receive academic credit, there needed 
to be coordination between the treatment center, John's parents, and the school. 
Secondly, the school might have information that would assist the treatment 
agency in breaking through John's denial, and/or information that would enhance 
John's self esteem. Thirdly, the SAT team began preparing for John's reentry 
back to school. 

The SAT team sent out a questionnaire form to all John's teachers and his 
wrestling coach. The questionnaire looked for indicators of possible problems 
and changes in attitudes and behaviors. Additionally, the SAT team gathered 
attendance records and grade reports. Upon gathering this information, the SAT 
team met and invited John's coach and two teachers to the staffing. It became 
apparent that John's perf'ormance was declining and there were indicators of 
changes in attitude, especially a notice of increased defiance to authority and 
noncompliance. Additionally, the coach admitted that he knew instances when 
John was drinking after wrestling matches, had found a bong among John's 
athletic things, and had suspected John of being under the influence at other 
times. 



^ Western Regional Center for Drug-Free Schools and Communities 

C^9^. 5-13 . o 



ERIC 



Section 6 

Evaluation Resources 



Section 6 



Evaluation Resource Notebook 



SECTION 6: EVALUATION RESOURCES 



Resources on AOD program evaluation have grown in the past few years. This section provides 
references to additional readings and resources. It includes: 

1. An annotated bibliography of a sampling of evaluation manuals and handbooks. 

2. A list of journals where articles and reports on alcohol and other drug programs as 
well as on program evaluation may be found. 

3. Western Regional Center publications on evaluation. 

4. Information and resource centers available to educators in the western region. 
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ANNOTATED BIBLIOGRAPHY OF EVALUATION MANUALS AND 

HANDBOOKS 



The references described below ^e not intended to be inclusive of the many fine works in the 
evaluation field. Rather they rej^esent a sample of materials that are easily available, 
inexpensive or free, and targeted to the persons with little or no background in evaluation. 

Clifford, M.A., & Davis, M. (1991). Evaluation tools for student assistance programs. Boulder, 
CO: National Organization of Student Assistance Programs and Professionals. 

The Evaluation Tools for Student Assistance Programs manual discusses what needs to be 
considered before starting an evaluation; lists the steps involved in evaluating S APs; and 
includes 21 evaluation tools. Sections discussing purpose, administration protocol and results 
interpretation are included for each instrument. 

Hawkins, ID., & Nederhood, B. (1987). Handbook for evaluating drug and alcohol prevention 
programs. Rockville, MD: U.S. Department of Health and Human Services. 

This handbook offers a step-by-step approach for the non-expert to evaluate school or 
community-based prevention programs. Each step offers specific details and offers examples 
from school or community programs. The evaluation model suggests that there are evaluation 
questions that need to be asked about effort, effectiveness, and efficiency of the program. Once 
the evaluation questions are determined, evaluation design, identifying outcome measures, 
organizing and collecting the data, elementary data analysis, and reporting the findings follow. 
Each topic is represented in the chapters (steps). 

Herman, J. L. (Ed.). (1987). Program evaluation kit. Newbury Park, CA: SAGE Publications. 

This series includes nine handy paperback books as follows: Evaluator*s Handbook, How to 
Focus an Evaluation, How to Design a Program Evaluation, How to Use Qualitative Methods in 
Evaluation, How to Assess Program Implementation, How to Measure Attitudes, How to 
Measure Performance and Use Tests, How to Analyze Data and How to Communicate 
Evaluation Findings. A step-by-step approach with illustrations is used in each of the 
handbooks. 

lOX Assessment Associates. (1988). Program evaluation handbook: alcohol abuse education: 
Program evaluation handbook: drug abuse education; Program evaluation handbook: 
smoking abuse education, Los Angeles, CA: lOX Assessment Associates. 

These are a series of three handbooks developed for the Center for Health Promotion and 
Education, Office of Disease Prevention and Health Promotion, U.S. Department of Health and 
Human Services. The handbooks are based on preliminary goals or standards of behavior, 
knowledge, skill, and affective outcomes in each content area. Panels of experts identified 
important outcomes for each of the content areas (alcohol education, drug abuse education, and 
smoking education). Based on these outcomes, a series of criterion-referenced instruments was 
developed and included in each manual. These instruments are considered useful for adults and 
school-aged children. 

Each manual begins with an overview of the content area. A chapter on evaluation essentials 
provides the novice with an overview of the evaluation issues. The bulk of the manual is filled 
with brief assessment instruments. While the instruments appear to have great face validity and 
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sometimes mirror questions used in national surveys, the reader is cautioned that the instruments 
have only been pilot-tested in small-scale tryouts and "have not been subjected to a formal 
empirical appraisal of their technical adequacy/' 

Joint Committee on Standards for Educational Evaluation. (1981). Standards for evaluations of 
educational programs, projects, and materials. New York: McGraw-Hill. 

A joint committee representing the American Educational Research Association, the American 
Psychological Association, and the National Council on Measurement in Education prepared 
thirty evaluation standards. The standards represent issues such as deciding whether you are 
ready to evaluate, the ethics surrounding evaluations, defining the evaluation problem, designing 
the evaluation, budgeting the evaluation, collecting and analyzing data, and reporting the 
evaluation.. 

Each standard is presented, followed by an overview, guidelines, pitfall^, caveats, illustrative 
case, and analysis of the case. It is an excellent overview for the beginning evaluator. 

Linney, J. A., & Wandersman, A. (1991). Prevention plus IIL Rockville, MD: U.S. Department 
of Health and Human Services. 

This is the third in a series of Prevention Plus publications from the Office for Substance Abuse 
Prevention (OSAP). It offers "up-to-date tools and techniques for alcohol and other drug (AOD) 
prevention planning and implementation." Both outcome and process evaluations are discussed 
and tools are provided for both. Instruments contained in the manual may be reproduced without 
permission from OSAP or the authors. 

Moberg, P. D. (1984). Evaluation of prevention programs: a basic guide for practitioners. 
Madison, WI: Wisconsin Clearinghouse. 

This 29-page guide covers uses of evaluation, planning a program evaluation, program 
monitoring and process evaluation, outcome evaluation and expectations for evaluation. The 
guide offers concise definitions, checklists and examples to guide the reader in conducting 
evaluations. It also includes examples of contact sheets described in the guide. 

National Crime Prevention Council. (1986). What, me evaluate? Washington, D.C.: National 
Crime Prevention Council. 

Written for the crime prevention practitioner, the handbook emphasizes a lay person's 
perspective toward evaluation. It explains why evaluation is not just for experts and offers 
concrete ways to document progress and success. Included in the handbook are four survey 
questionnaires that can be used to assess citizens' attitudes and beliefs about crime and crime 
prevention. 

Worthen, B.R., & Sanders, J.R. (1987). Educational evaluation: alternative approaches and 
practical guidelines. New York: Longman. 

Although designed as a basic text for courses on evaluation, the book serves as a useful reference 
for readers looking for a comprehensive overview of educational evaluation. The book clarifies 
basic concepts in evaluation, describes a variety of alternative approaches, and provides detailed 
suggestions in planning, conducting, and using evaluations. 
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JOURNALS ON ALCOHOL AND OTHER DRUG PREVENTION AND EDUCATION 

There are no journals yet dedicated to dissemination of evaluation reports in the AOD field, so 
articles on evaluating AOD programs may appear in an> of these journals: 



Addictive Behaviors (6/year) 
Pergamon Press, Inc. 
Journals Division 
Maxwell House 
Fairview Park 
Elmsford,NY 10523 
TEL: 914-592-7700 

AmericaD Journal of Drug and Alcohol Abuse 
(4/year) 

Marcel Dekker Journals 
270 Madison Avenue 
New York, NY 10016 
TEL: 212-696-9000 

American Journal of Orthopsychiatry 
(quarterly) 

American Orthopsychiatric Association, Inc. 

19 W. 44ih St. 

New York, NY 10036 

TEL: 212-354-5770 

American Journal of Public Health (monthly) 
American Public Health Association 
1015 15lhSt.,NW 
Washington, D.C. 2005 
TEL: 202-789-5600 

Health Education Quarterly (quarterly) 

John Wiley & Sons. Inc. 

Periodicals Division 

605 Third Avenue 

New York, NY 10158-0012 

TEL: 212-692-6000 

International Journal of the Addictions 
(12/year) 

Marcel Dekker Journals 
270 Madison Av ..me 
New York, NY 10016 
TEL: 212-696-9000 



Journal of Addictive Diseases (quarterly) 
The Hawonh Press, Inc. 
10 Alice Street 
Binghamion, NY 13904 
TEL: 800-3-HAWORTH 

* Journal of Adolescent Chemical 
Dependency (quarterly) 

The Hawonh Press, Inc. 
10 Alice Street 
Binghamion, NY 13904 
TEL: 800-3-HAWORTH 

Journal of Adolescent Health Care (6/year) 
(Society for Adolescent Medicine) 
Eisevier Science Publishing Co,, 
655 Avenue of the Americas 
New York, NY 10010 
TEL: 212-989-5800 

Journal of Adolescent Research (quarterly) 
Sage Publications, Inc. 
2455 Teller Road 
Newbury Park, CA 91320 
TEL: 805-499-0721 

* Journal of Alcohol and Drug Education 

(3/year) 

Alcohol & Drug Problems Assn of No. America 
1120 East Oakland 
P.O. Box 100212 
Lansing. ME 48901 
TEL: 517^84-0016 

Journal of American College Health 

(bimonthly) 

(American College Health Association) 
Heldref Publications 
4000 AlbermarleSt.,NW 
Washington, D.C.. 20016 
TEL: 202-362-6445 



JAMA: Journal of the American Medical 

Association (weekly) 

American Medical Association 

515 N. State St. 

Chicago, IL 60610 

TEL: 312464-0183 



Journal of Applied Psychology (bimonthly) 
American Psychological Association 
1200 17th St., NW 
Washington, D.C. 20036 
TEL: 202-955-7600 
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Journal of Child Psychology & Psychiatry & 

Applied Disciplines (7/year) 

(Assn of Child Psychology and Psychiatry) 

Journals Division Pcrgamon Press, Inc. 

Maxwell House, Fairview Park 

Elm$ford,NY 10523 

TEL: 914-592-7700 

Journal of Consulting and Clinical 
Psychology (bimonthly) 
American Psychological Association 
1200 17th St.,NW 
Washington, D.C. 20036 
TEL: 202-955-7600 

Journal of Counseling & Development 
(bimonthly) 

American Assn for Counseling and Development 
5999 Stevenson Avenue 
Alexandria, VA 22304 
TEL: 703-823-9800 

• Journal of Drug Education (quarterly) 
Baywood Publishing Co.» Inc. 

26 Austin A^venue 
P.O. Box 337 
AmityviIle,NY 11701 

• Journal of Drug Issues (quarterly) 
Journal of Drug Issues, Inc. 

P.O. Box 4021 
Tallahassee, FL 32315-4021 



• Journal of School Health (10/year) 
American School Health Association 
7263 State Route 43 
P.O. Box 708 
Kent, OH 44240 

Journal of Studies on Alcohol (bimonthly) 
(Rutgers Center of Alcohol Studies) 
Alcohol Research Documentation, Inc. 
Box 969 

Piscalaway, NJ 08855 
TEL: 908-932-2190 

Journal of Substance Abuse (quarterly) 
Ablex Publishing Coiporation 
355 Chestnut St 
Norwood, NJ 07648 
TEL: 201-767-8450 

Preventive Medicine (bimonthly) 

(American Health Foundation) 

Academic Press, Inc. 

Journal Division 

1250 Sixth Avenue 

San Diego, CA 92101 

TEL: 619-230-1840 

Psychological Reports (bimonthly) 

Dr. C.H. Ammons & Dr. R.B. Ammons, Eds. & 

Pubs. 

Box 9229 

Missoula, MT 59807 



Journal of Public Health Policy (quarterly) 
Journal of Public Health Policy, Inc. 
208 Meadowood Drive 
South Burlington, VT 05403 
TCL: 802-658-0136 



Recent issues of these joumais are available for use at the Western Regional Center 
for Drug-Free Schools and Communides at 101 SW Main, Portland, Oregon 
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JOURNALS ON PROGRAM EVALUATION 

Although the focus of these journals is evaluation rather than on evaluation of ADD 
programs, these journals provide discussions of current thinking and ideas in the field of 
evaluation. 

Educational Evaluation and Policy Analysis (quarterly) 

American Educational Research Association (AERA) 
1230 17th Street, NW 
Washington, D.C 20036-3078 

Evaluation and Program Planning (quarterly) 

Pergamon Press, Inc. 
Journal Division 
Maxwell House 
Fairview Park 
Elmsford.NY 10523 
HEL: 914-592-7700 
FAX: 914-592-3625 

Evaluation Re>iew (bimonthly) 

Sage Publications, Inc. 
2455 TeUer Road 
Newbury Park, CA 91320 

Evaluation Studies Review Annuals (annually) 

Sage Publications, Inc. 
2455 TeUer Road 
Newbury Park, C A 91320 

New Directions for Program Evaluation (quarterly) 

(American Evaluation Association) 
Jossey-Bates Inc., Publishers 
350 Sansome St.. 5th Floor 
San Francisco, C A 94104 
TEL: 415433-1767 
FAX: 415-433-0499 

Studies in Educational Evaluation (3/year) 

(UCLA Center for the Study of Evaluation) 

Pergamon Press, inc. 

Journals Division 

Maxwell House 

Fairview Park 

Elmsford.NY 10523 

TEL: 914-592-7700 

FAX: 914-592-3625 
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WESTERN REGIONAL CENTER PUBLICATIONS ON 

EVALUATION 



Gabriel, R.M., & Brinkerhoff, C. (1990). Developing a community profile: A handbook 
for using pre-existing data in prevention planning. Portland, OR: Northwest 
Regional Educational Laboratory. 

Pollard, J.A., Gabriel, R.M., & Arter, J.A. (1991). Surveys of student alcohol and other 
drug use: A consumer's guide (2nd ed.) Portland, OR: Northwest Regional 
Educational/Laboratory. 

Regional Drug Initiative. (1991). Drug impact index, 2nd edition. Portland, OR: 
Northwest Regional Educational Laboratory. 

Western Regional Center for Drug-Free Schools and Communities. (1991). Alcohol and 
other drug programs: Abstracts of selected evaluation studies. Portland, OR: 
Northwest Regional Educational Laboratory. 

Western Regional Center for Drug-Free Schools and Communities. (1991). Alcohol and 
other drug program evaluation at the local level: A pilot site approach. Portiand, 
OR: Northwest Regional Educational Laboratory. 

Yen, F.B. (1992). Evaluating AOD Program Implementation, Portland, OR: Northwest 
Regional Educational Laboratory. 
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INFORMATION AND RESOURCE CENTERS 

A wealth of information on a variety of topics is available from the following national 
and regional centers. Check with these sources for updates on resources in evaluating 
AOD programs. 

1. OSAFs National Clearinghouse for Alcohol and Drug Information 
(ONCADI) 

P.O. Box 2345 

Rockville, MD 20847-2345 

1-800-729-6686 

Funded by the U.S. Department of Health and Human Services, ONCADI 
provides searches from its database on specific alcohol and other drug 
topics at no cost. It also disseminates a bimonthly publication titled: The 
OSAP Prevention Pipeline which contains prevention news, new program 
resources available from the Office of Substance Abuse Prevention and 
other federal agencies, as well as abstracts of recent journal articles and 
books in the alcohol and other drug field. The Western Regional Center 
for Drug-Free Schools and Communities in Portland, Oregon, subscribes 
to The OSAP Prevention Pipeline, 

2. Educational Resources Information Center (ERIC) 
Dept. CCE 

1600 Research Boulevard 
Rockville, MD 20850 

Supported by the U.S. Department of Education, Office of Educational 
Research and Improvement (OERI), ERIC consists of 16 clearinghouses 
that respond to inquiries about its database, which is accesible to the 
public at more than 3,000 locations worldwide. Contact the 
Clearinghouse on Tests, Measurements, and Evaluation through the above 
toll-free number for citations on journal articles and documents related to 
educational evaluation. 

3. Western Regional Center for Drug-Free School and Communities 

Judith A. Johnson, Director 

A. Training and technical assistance on evaluation is available at no cost to 
school districts in the west. For further information contact the Western 
Regional Center office listed below that serves your state. 

Alaska, Idaho, Montana, Oregon, Washington, and Wyoming: 

Carlos Sundermann/Flora Yen 

Northwest Regional Educational Laboratory 

101 SW Main Street, Suite 500 

Portland, OR 97204 

503-275-9478 or 9497 

1-800-547-6339, ext. 478 or 497 
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Hawaii and the Pacific Islands: 

Harvey Lee 

Honolulu Field Office 

1 164 Bishop Street, Suite 1409 

Honolulu, HI 96813 

808-532-1904 

Northern California and Northern Nevada: 

Ralph Baker 

Far West Laboratory for Educational 

Research and Development 

730 Harrison Street 

San Francisco, CA 94107-1242 

415-565-3000 

Southern California and Southern Nevada: 

Carol R Thomas 

Southwest Regional Laboratory 

4665 Lampson Avenue 

Los Alamitos, California 90720 

310-598-7661 

B. Resource Center Services 

The Western Regional Center's main resource collection is housed in 
Portland, Oregon. The following types of services are offered: 

Library: A collection of AOD prevention curricula, programs, 
monographs, and videos is available free for a three week loan period. 
Telephone your requests to the Western Center or visit the Center, 8:30 - 
5:00 p.m., 503-275-9486 or 1-800-547-6339, ext. 486. 

Western Center Publications: The Center disseminates over 30 free 
publications to the public. These include the Western Center quarterly 
newsletter (mailed to over 16,000 clients). Prevention Research Updates 
(a publication that summarizes the recent research on AOD use and its 
prevention), and other publications of interest to preventionists on such 
topics as peer education, fetal effects of parental AOD use, summaries of 
successful prevention programs, and guides to student AOD surveys and 
parenting skills curricula. A listing of these publications is available upon 
request. 

Free Materials: The Center is a Regional Alcohol and Drug Awareness 
Resource (RADAR) Network member and thus has the current 
publications available through OSAP's National Clearinghouse for 
Alcohol and Drug Information (ONCADI). Other free materials include 
pamphlets and brochures from local agencies and programs. 

Organization files: Files with catalogs, brochures, and program 
information are kept on organizations, publishers, agencies and 
individuals. 
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Subject Files: Current newspaper, magazine, joumaU and research 
articles are collected for reference use on a wide range of alcohol and 
other drug related topics. 

Newsletters, Journals: The Center subscribes to 50 newsletters/journals. 
These can be used by the public at the Center, 

Satellite resource offices are located in Honolulu (Hawaii), Los Alamitos 
(California), and San Francisco at the addresses noted above. 

4, Other Sources 

Check with your school district administration, statewide AOD 
coordinators, as well as educational or psychological departments in 
nearby universities and colleges for possible assistance on evaluation. 
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EVALUATION INSTRUMENTS 

Samples of two instruments are included in this section. The Self-Assessment Instrument for 
School Programs can be used by school and district teams to review program growth and to 
pinpoint areas for further development. The instrument has also been used to encourage broad 
participation in goal setting. The second instrument, the Student Alcohol and Drug Use Survey 
(short form), is a condensed version of a longer questionnaire that has been administered to 
students at grades six, eight, ten, and twelve in two Western states - Hawaii and Washington. 
When completed anonymously by students, the short form of the questionnaire is just as useful 
in providing a gauge of the extent and scope of AOD use in a school or district. 
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SELF-ASSESSMENT INSTRUMENT FOR SCHOOL PROGRAMS 
IN ALCOHOL AND OTHER DRUG PREVENTION 

A PROGRAM IMPROVEMENT EVALUATION AID 



Western Regional Center For Drug-free Schools And Communities 
Northwest Regional Educational Laboratory 
Portland, Oregon 97204 



Instructions for Using the Self-Assessment Instrument 



The Western Regional Center School Program Self-Assessment Instrument is an informal tool 
used to assist districts in identifying areas for additional development in their alcohol and other 
drug education programs. A comprehensive program consists of nine categories. Each 
component is assessed by rating a series of indicators. Ratings may be totaled or averaged for 
each component to provide an indication of specific program practices that may need 
improvement or greater emphasis. 

The Self-Assessment Instrument is a measure of self-reported perceptions, and is neither a guide 
for structured observation nor a substitute for a program evaluation. In areas in which self- 
reports arc often inaccurate (e.g.> use of time, communication of expectations), the assessment 
should be supported by more objective measures. 

Procedures for Use 

1. Encourage Broad Participation 

The self- assessment process has value beyond the information it provides. It facilitates 
staff participation in goal setting; and the list of important practices is likely to stimulate 
learning, reflection, and debate. We suggest that all members of the school team 
complete the instrument; including teachers, principals, counselors, and central office 
administrators. 

2. Explain Purpose 

Staff members need to understand that the purpose of the assessment process is to 
identify areas for program development, and not to assess personal performance or to rate 
the overall program. 

3. Summarize completed ratings for each component. 

Each rater should rate each item, and compute an average rating for each component. 

If individuals feel they have no basis for rating certain components, they may leave them 
blank. However, it is important to respond to every item within a component. (If items 
are omitted, assign them the average of the rated items in that section.) 

4. Share Results and Select Target Areas for Improvement 
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The preliminary results of the self-assessment provide a shared understanding of the 
status of the program in terms of several important implementition indicators. It can also 
provide a stimulus for selecting target areas for program improverrient. It is important 
that the target-setting process involve discussion, active participation^ and consensus 
building among school team members, and not depend on mechanical averaging of the 
assessment results. 



The results of the Self- Assessment Instrument are only one source of data to consider in 
targeting areas for development. Outcome data and other evaluative information, perceived 
ncedSt classroom observation, and ongoing district or school improvement activities may 
influence staff to adjust the selection of target areas. Development efforts should focus on an 
area that is perceived as both important and capable of change. 
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SCHOOL PROGRAM SELF-ASSESSMENT INSTRUMENT 



Not at ail 
like our 
program 



Very much 
like our 
program 



A, SCHOOL TEAM INVOLVEMENT 

1. The school team has a range of 
representation from school 
leadership to community members. 

2. The school administration has given 
the team the time and resources to 
complete the planning process. 

3. A school team leader has been 
selected and understands his/her role. 

4. The school team leader is able to 
assess team interests and abilities 
and make effective and efficient 
assignments. 

5. School team members feel 
comfortable in working through the 
process with other team members. 



2 
2 



3 
3 



4 
4 



5 
5 



B. PARENT/COMMUNITY INVOLVEMENT 

1. Parents are included in the needs 
assessment and planning process. 

2. Community members are included in 
the planning and implementation. 

3. Community involvement includes a 
broad representation of agencies 
such as law enforcement, health and 
social agencies, business and 
religious organizations. 

4. Parent education is available for 
parents at all grade ranges. 

5. Parents generally are supportive of 
the alcohol and other drug 
prevention programming (there is no 
formal or informal opposition). 



3 4 



3 4 



3 4 



C. NEEDS ASSESSMENT 
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SCHOOL PROGRAM SELF-ASSESSMENT INSTRUMENT 



Not at all 
like our 
program 



Very much 
like our 
program 



1. Needs assessment information is 
gathered on an ongoing basis. 

2. A survey of student alcohol and 
other drug use has been conducted 
within the last two years. 

3. A survey of prevention resources and 
activities in the school and 
community has been conducted. 

4. Data collected are used in planning a 
comprehensive alcohol and other 
drug education program. 



D. PROGRAM PLANNING 

1. Clear goals and objectives have been 
established for all elements of a 
comprehensive drug-free schools 
plan. 

2. Goals and objectives have been 
presented to and approved by 
school/district administration. 

3. Staff assignments to carry out the 
objectives have been approved by 
the school/district administration. 

4. Goals and objectives have been 
shared with the entire school staff 
and supported by them. 



E. POLICY DEVELOPMENT 

1 . Clear and consistent policies related 
to alcohol and other drug use exist at 
school and district levels. 
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SCHOOL PROGRAM SELF-ASSESSMENT INSTRUMENT 



Not at ali 
like our 
program 



Very much 
like our 
program 



2. Existing policies deal with all 
important issues relating to alcohol 
and other drug prevention: clear 
definition of what constitutes an 
offense, consequences of violations 
of policies, consistency with state 
and local laws, etc. 

3. Comprehensive policies exist for all 
areas of student assistance programs: 
early intervention, referral and 
aftercare, 

4. Policies are known by staff, students, 
parents, and community members. 



1 2 3 4 5 



1 2 3 4 5 



1 2 3 4 5 



F. IMPLEMENTATION OF PREVENTION CURRICULUM AND 
ACTIVrriES 



L Prevention goals and objectives 
include curriculum from K-12 as 
well as other prevention activities, 

2. Teachers and staff involved in 
programming have ample 
opportunity for and participate in 
initial and ongoing training, 

3, Training opportunities are taker as a 
sequence of related activities for a 
comprehensive program and are not 
viewed as "one shot" offerings, 

4, The curriculum and program 
activities are being implemented as 
intended, 

5. There are sufficient numbers of 
instructional materials and program 
kits available. 



1 



G, IMPLEMENTATION OF ASSISTANCE PROGRAMS 

1, A student assistance program is in 1 i 

place at all levels: elementary, 
middle, and secondary schools. 
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SCHOOL PROGRAM SELF-ASSESSMENT INSTRUMENT 



Not at all 
like our 
program 



Very much 
like our 
program 



2. Expectations in the area of 
intervention are clear to staff, 
students and parents and are based 
on policy. 

3. Ongoing supervision and feedback is 
provided to teachers and staff in the 
school. 

4. Inservice training topics are 
identified and training is made 
available on an ongoing basis. 

5. Cooperative relationships with 
community resources have been 
developed and documented. 

6. There is an employee assistance 
program in place. 



1 2 3 4 5 



1 2 3 4 5 



1 2 3 4 5 



1 2 3 4 5 



1 2 3 4 5 



H. EVALUATION 

1. An ongoing system of collection of 
evaluation information is in place. 

2. Both program implementation and 
outcome data are being collected. 

3. Data are used to modify or focus 
programs. 



L DISSEMINATION 

1. Key elements of the program are 
communicated to parents, 
community members, and the 
general public. 

2. These audiences get periodic reports 
or announcements of important 
program activities. 

3. Evaluation and needs assessment 
information is provided to the 
administration and the public. 



1 2 3 4 5 



1 2 3 4 5 



1 2 3 4 5 
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SCHOOL PROGRAM SELF-ASSESSMENT INSTRUMENT 

Not at all Very much 

like our like our 

program program 

4, Partnerships with community 1 2 3 4 5 

resources for cooperative 
dissemination campaigns are 
underway. 
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PROGRAM SELF-ASSESSMENT SUMMARY SHEET 



Total Total Percent Priority Rank 

Score Possible Of Total (Lowest Score 

For Possible is 1, Next is 

Area 2, etc.) 



# A. SCHOOL TEAM INVOLVEMENT 25 

B. PARENT/COMMUNTTY 

INVOLVEMENT 25 

C. NEEDS ASSESSMENT 20 

D. PROGRAM PLANNING 20 

E. POLICY DEVELOPMENT 20 

F. IMPLEMENTATION OF 

0 PREVENTION CURRICULUM _21 

G. IMPLEMENTATION OF 
ASSISTANCE PROGRAMS _2il 

H. EVALUATION _J1 

I. DISSEMINATION 20 



CATEGORY: 

^TEACHER 

^ADMINISTRATOR 

^COUNSELOR 

^OTHER 
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Student Alcohol and Drug Use Survey 

Please take a few minutes to complete this survey. The purpose of the survey is to find out 
how much students in this school use tobacco, alcohol, and other drugs. Your answers will help 
us understand the needs of students in our school. 

You are not required to complete this survey. If there is any question that would upset you or 
your parents, just leave it blank. 

Your answers are confidential. Do not write your name anywhere on this survey. An envelope 
will be passed around when you are finished so you can insert your survey without anyone 
seeing your answers. 

Make a check mark in the circle for the answer you choose. Check only one answer for each 
question. 

1. What is your current grade level? 
Q5 08 Oil 
0 6 0 9 Cl2 

O 7 O 10 O Ungraded 

2. What is your sex? 
Oa. Male 

O b. Female 

3. Are you aware of someone close to you who has a drinking or drug problem? 
O a. Yes, someone I live with 

O b. Yes, a friend 

G c. Yes, both someone I live with and a friend 
O d. No, no one close to me 

4. Of your closest friends, how many use alcohol once a month or more often? 
O a. None 

O b. A few 
O c. Some 
Od. Most 
O e. All 

5. How difficult do you think it would be for you to get marijuana (grass, pot, hash)? 
^ a. Probably impossible 

b. Very difficult 

c. Fairly difficult 

d. Fairly easy 

e. Very easy 

6. Is there drinking or drug use at most parties you attend? 
O a. Yes, alcohol 

Q b. Yes, drugs 

Q c. Yes, both alcohol and drugs 
Od. No 

O e. Don't attend parties 

Shori Form Western Regional Center 
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When did you last use each of the following drugs? 

7. Tobacco (smoking, chewing, snuff) 

8 a. Never used in my lifetime 
b. Used at least once, but not in the last 30 days 
O c. Used at least once in the last 30 days 

8. Alcohol (beer, wine, wine cooler, hard liquor) 
O a. Never used in my lifetime 

Q b. Used at least once, but not in the last 30 days 
O c. Used at least once in the last 30 days 

9. Marijuana (pot, grass, hash) 

O a. Never used in my lifetime 

O b. Used at least once, but not in the last 30 days 

O c. Used at least once in the last 30 days 

10. Cocaine (snow, coke, crack) 

O a. Never used in my lifetime 

O b. Used at least once, but not in the last 30 days 

O c. Used at least once in the last 30 days 

11. Inhaled something to get high (glue, gasoline, paint thinner, spray cans, white-out) 
O a. Never used in my lifetime 

O b. Used at least once, but not in the last 30 days 
O c. Used at least once in the last 30 days 

12. Anv other illegal drug (excluding alcohol, tobacco, marijuana, or cocaine) 

O a. Never used in my lifetime 

O b. Used at least once, but not in the last 30 days 

O c. Used at least once in the last 30 days 

The next two items concern your drinking. 

13. How old were you when you had your first full drink (a can of beer, a full glass of 
wine, or a mixed drink)? 
O a. 1 have never had a drink 

10 or younger 

1 1 or 12 
13 or 14 
15 or older 

14. How much do you usually drink at one time? 
O a. I don't drink 

Less than one can or glass of beer, wine, or mixed drink 
One can or glass of beer, wine, or mixed drink 
2-4 cans or glasses of beer, wine, or mixed drink 
5 or more cans or glasses of beer, wine, or mixed drink 

15. How honestly did you answer this survey? 
O a. Very honestly 
O b. Somewhat honestly 
O c. Dishonestly 
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Instructions for Administering and Analyzing 
the Student Alcohol and Drug Survey 



Rationale 



This student use survey is easy to administer and to analyze manually. If properly administered, 
this survey will provide valid, reliable data on student use. These survey results can serve as 
the basis of a prevention needs assessment and provide baseline data for assessing the overall 
impact of the prevention effort. This short instrument cannot, however, provide the extensive 
diagnostic information available from some other surveys. 

Specifically, the survey is designed to: 

0 Assess risk factors associated with student use 

o Determine the recency of use for several of the most frequently abused substances 
0 Distinguish between lifetime use and thirty-day use for comparison to national data 
0 Deter:nine use for each grade level 
0 Identify the extent of binge drinking 

Planning Survey Administration 

Parental notification. You must notify parents that you intend to administer the survey. That 
notification can take several forms, but a letter to parents is preferred. The notice should 
emphasize that the information will be used to help plan or improve your prevention and that 
the survey is voluntary and completely confidential. 

Sampling. A uniform procedure is needed for selecting students that will participate in the 
survey. You will want to replicate this procedure when the survey is administered again. At 
least one grade in every school, including elementary schools, should be surveyed. The reading 
level of the survey, however, makes it most appropriate for grades 6 and up. Most districts • 
elect to administer the survey to all students at the selected grade levels. If students are 
sampled within each grade, the sampling must be random. Administration to selected 
classrooms does not qualify as a random sample. 

Staff training. The climate established by the person administering the survey is important in 
determining how seriously the students take the survey. A brief meeting, formal or informal, is 
helpful in clarifying the purpose for the survey as well as procedures for administration. 

Administration. The survey should be administered during the same period throughout the 
school so students do not have a chance to discuss the survey before responding to it. 
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Analyzing Student Responses 



1. Prepare for the analysis. 

a. Separate the completed surveys into piles by grade. 

b. Screen the sheets in each pile and remove those that are blank or inappropriate. 

Count the student as refusing to complete the survey if all or nearly all 
the items are left blank. 

Remove the answer sheet if the student answered "Dishonestly" on the last 
item. 

Remove the answer sheet if the student wrote in inappropriate responses 
(e.g. wrote in "Other" for Sex). 

2. Complete Part I of the Summary Worksheet for each grade. 

3. Tally responses to each item on the Response Tally Worksheet. 

a. Mark the responses from each survey in the proper column as shown in the 
example. 

b. Add columns A-E to find the total number of students answering the iteni. 

4. Calculate percentages using Part II of the Summary Worksheet 

a. For each item, divide columns A through E by the Total from the Response 
Tally Worksheet to obtain the proportion of students choosing that response. 

b. Multiply by 100 to obtain a percentage. For most audiences you may want to 
round this percentage to the nearest whole number. 

c. Enter the percentages for each item in the corresponding column in Part 11. 
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Summary Worksheet 
for Student Use Survey Results 

Part L Administration 

/ / Date survey was administered 

Grade level summarized (only one per Summary Worksheet) 

Number of students absent when survey was administered 

Number of students refusing to complete survey 

Number of surveys removed due to inappropriate responses 

Part IL Computation of Percentages 

A. B. c a E. 

Example ~ 

2. Gender 

3. Someone close 

4. Friend's use 

5. Marijuana access 

6. Use at parties 

7. Tobacco 

8. Alcohol 

9. Marijuana 

10. Cocaine 

11. Inhalants 

12. Other drugs 

13. First drink 

14. Binge drinking 

15. Honesty 
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Response Tally Worksheet 



Grade level (only one per worksheet): 




2. Gender 

3. Someone close 

4. Friend's use 

5. Marijuana access 



6. Use at parties 



7. Tobacco 



8. Alcohol 



9. Marijuana 



10. Cocaine 



11. Inhalants 



12. Other drugs 



13. First drink 



14. Binge drinking 



15. Honesty 
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